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ABSTRACT 


In  this  thesis,  the  relationship  between  the  characteristics  or  attributes  of  a 
military  weapon  system  (e.g.,  speed,  reliability,  survivability)  and  the  effectiveness 
of  that  system  is  thoroughly  examined.  Success  in  system  acquisition  relies  on  (1) 
the  early  identification  and  successful  incorporation  of  those  system  attributes  that  are 
critical  to  system  effectiveness,  and  (2)  the  specification  of  numerical  values  for  the 
system  attributes  (the  system  requirements)  that  maximizes  system  effectiveness  at 
an  acceptable  cost.  New  definitions  for  system,  system  attributes,  and  system 
effectiveness,  as  well  as  relevant  DoDI  5000.2  guidance  are  provided.  In  addition  to 
the  currently-mandated  battle  level  at  which  system  effectiveness  should  be  measured 
(in  terms  of  engagement  or  battle  outcomes),  the  author  uses  a  wide  spectrum  of 
system  acquisition-related  literature  to  advocate  that  system  effectiveness  should  also 
be  measured  at  the  mission  level  (in  terms  of  mission  outcome).  Several  existing 
mathematical  models  which  combine  a  few  key  system  attribute  measurements  into 
single-number  measures  of  system  effectiveness  in  accomplishing  a  particular  mission 
are  described.  Then,  the  author  proposes  a  hierarchy  or  tree  which  relates  many 
system  attributes  to  the  four  key  attributes,  Availability,  Reliability,  Survivability,  and 
Capability,  and  hence  to  system  effectiveness  in  accomplishing  a  specified  mission. 
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I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  examine  the  relationship  between  the 
characteristics  or  attributes  of  a  military  weapon  system  and  the  effectiveness  of  the 
system.  System  effectiveness  is  widely  viewed  as  the  result  of  the  complex 
interaction  of  a  multitude  of  system  attributes,  such  as  lethality,  reliability, 
survivability,  and  maintainability.  The  only  given  official  guidance  for  weapon 
system  acquisition  regarding  system  effectiveness  measurement  is  that  the  measures 
of  effectiveness  (MOEs)  should  measure  operational  capabilities  in  terms  of 
engagement  or  battle  outcomes.  Because  the  official  guidance  does  not  provide  a 
standard  model  for  system  effectiveness  measurement,  several  models  have  been 
offered  which  reduce  the  measurement  of  system  effectiveness  to  an  equation 
involving  the  combination  of  a  few  key  attribute  measurements,  or  performance 
parameters. 

In  this  thesis,  the  author  will  propose  a  mission  level  for  measuring  system 
effectiveness,  in  addition  to  the  battle  level.  Based  on  this  alternative  level,  and  a 
review  of  several  existing  system  effectiveness  models,  the  author  will  propose  a 
hierarchy  of  system  attributes  which  may  affect  overall  system  effectiveness  in 
accomplishing  a  specified  mission.  Ultimately,  this  thesis  should  provide  the  reader 
with  a  better  understanding  of  the  relationship  between  system  attributes  and  system 
effectiveness. 

B.  APPROACH 

The  author  will  accomplish  the  purpose  described  above  through  the  following 
chapters.  First,  in  Chapter  II,  a  few  key  definitions  and  issues  will  be  addressed.  In 
Chapter  III,  the  current  system  effectiveness  measurement  process,  as  defined  in  the 
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official  U.S.  Code  and  DoD  guidance,  will  be  presented.  In  Chapter  IV,  a  mission 
level  for  measuring  system  effectiveness  will  be  proposed  which  allows  a  linkage 
between  system  attributes  and  system  effectiveness.  In  Chapter  V,  the  relationship 
between  system  attributes  and  system  effectiveness  in  the  system  design/test  and 
evaluation  processes  will  be  examined.  In  Chapter  VI,  the  quantification  of  system 
effectiveness  will  be  examined,  and  two  general  methods  of  combining  key  system 
attributes  into  single-number  system  effectiveness  measurements  will  be  presented, 
along  with  several  representative  models.  Las!1  in  Chapter  VII,  the  author  will 
propose  a  hierarchy  or  tree  of  system  attributes  which  may  affect  overall  system 
effectiveness  in  accomplishing  a  specified  mission. 

C.  SCOPE 

This  thesis  is  limited  to  the  standard  U.S.  acquisition  process  for  weapons 
systems  which  requires  all  of  the  DoD  5000-mandated  documentation.  It  will  not 
consider  recent  relaxations  in  documentation  requirements  for  some  minor  systems. 
Because  the  DoD  5000  documents  are  currently  being  revised,  those  draft  revisions 
that  are  pertinent  to  this  thesis  are  included.  As  a  further  limitation,  the  included 
models  and  systems  are  only  samples  of  a  model  population  that  cannot  be  fully 
included  in  one  report.  Therefore,  there  are  undoubtedly  some  aspects  of  system 
effectiveness  measurement  which  will  not  be  covered.  This  thesis  will,  however, 
attempt  to  address  many  key  areas  which  are  applicable  to  the  subject  matter. 
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II.  SYSTEMS,  ATTRIBUTES,  AND  EFFECTIVENESS 


A.  INTRODUCTION 

In  this  chapter,  the  author  will  provide  some  preliminary  information  and 
definitions  as  a  foundation  for  the  follow-on  examination  of  system  effectiveness 
concepts  and  measurements.  First,  some  basic  definitions  for  "system"  and  "system 
attributes"  will  be  provided.  Next,  the  author  will  preview  the  examination  of  system 
effectiveness  by  presenting  a  list  of  the  key  issues  which  will  be  addressed  in  this 
thesis. 

B.  KEY  DEFINITIONS 
1.  System 

A  specific  definition  of  what  constitutes  a  system  has  yet  to  be  universally 
accepted.  Several  different  definitions  of  systems  can  be  found  in  the  system 
effectiveness-related  literature.  Three  examples  are: 

A  set  of  interacting  components  composed  of  humans  and  machines 
(including  software)  directed  toward  performing  a  function  or  number 
of  functions  and  operating  within  the  constraints  of  time  and  specified 
environments.  [Ref.  1  :p.  6] 

A  number  of  parts  which  are  connected  together  in  order  to  transform 
a  given  set  of  inputs  into  a  given  set  of  outputs.  [Ref.  1  :p.  81  ] 

A  set  of  interrelated  components  which  interact  with  one  another  in  an 
organized  fashion  toward  a  common  purpose.  [Ref.  2:p.  3] 

This  thesis  will  focus  on  military  weapon  systems,  which  are  defined  by  the 
author  using  a  variation  of  the  first  definition  given  in  this  section: 
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"A  set  of  interacting  components  composed  of  humans  and  machines 
assembled  for  the  purpose  of  one  or  more  military  missions  and  operating  within  the 
constraints  of  time  and  specified  environments." 

2.  System  Attributes 

As  there  is  no  universally-accepted  definition  for  a  system,  there  is  also  no 
universally-accepted  term  used  to  describe  the  attributes  of  a  system.  In  this  report, 
the  author  will  combine  the  most  commonly-used  terms  into  the  following  definition 
for  system  attributes:  "The  characteristics,  capabilities,  or  parameters  of  a  system." 
Examples  of  some  possible  system  attributes  are  shown  in  Table  2.1. 


Speed 

Survivability 

Availability 

Range 

Susceptibility 

Maintainability 

Payload 

Vulnerability 

Logistic  Supportability 

Weight 

Compatability 

Manpower  Supportability 

Diameter 

Human  Factors 

Transportability 

Maneuverability 

Safety 

Reliability 

Lethality 

Interoperability 

Natural  Environmental 
Effects/Impacts 

Table  2.1.  Some  Possible  System  Attributes 


Note  that  many  of  the  attributes  listed  in  Table  2.1  can  easily  be  quantified  at 
the  beginning  of  a  system  acquisition  program  using  well  established  metrics.  For 
example,  the  attributes  of  speed,  range,  and  payload  can  have  the  metrics  of  kilo¬ 
meters/hour,  kilometers,  and  kilograms,  respectively.  These  quantifiable  attributes 
of  the  developing  or  developed  system  can  also  be  tested  and  evaluated  relatively 
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easily  because  they  are  essentially  deterministic  in  nature;  that  is,  a  relatively  small 
number  of  non-destructive  tests  is  sufficient  to  obtain  an  estimate  of  their  value. 

Other  attributes,  such  as  reliability,  safety,  and  survivability,  are  more  difficult 
to  quantify  at  the  beginning  of  system  acquisition  because  of  their  random  nature.  For 
these  attributes,  probability  measures,  such  as  mean  time  between  failures,  probability 
of  a  mishap  in  1 00,000  flight  hours,  and  probability  of  mission  survival,  could  be  used 
as  the  defining  metric.  The  testing  required  to  determine  these  attributes  is  also  more 
difficult  because  of  the  relatively  large  number  of  tests  required  to  obtain  an  accurate 
estimate  of  their  value.  In  addition,  the  system  testing  may  be  complicated  by  the 
destructive  nature  of  the  attribute  in  the  event  of  failure,  as  in  the  case  of  safety  and 
survivability. 

A  third  type  of  attribute,  such  as  interoperability  and  compatability,  can  not  be 
quantified.  These  attributes  are  of  a  "yes  or  no"  nature;  either  the  system  is 
interoperable  or  compatible  with  another  system  in  one  aspect  or  another,  or  it  is  not. 
However,  even  with  these  attributes,  there  may  be  other  attributes  that,  when 
quantified,  will  result  in  a  positive  yes.  For  example,  a  radio  being  developed  may 
be  compatible  with  an  existing  radio  if  the  frequency  bands,  which  are  quantifiable, 
are  the  same. 

The  crux  of  a  successful  system  acquisition  program  is  (1)  early  identification 
and  successful  incorporation  of  those  attributes  that  are  critical  to  system 
effectiveness  and  (2)  the  specification  of  numerical  values  for  the  system  attributes 
(the  system  requirements)  that  maximizes  system  effectiveness  at  an  acceptable  cost. 

3.  Summary 

Usage  of  the  term,  "attribute",  in  this  report  is  a  conscious  attempt  at 
uniformity.  In  the  official  DoD  guidance  (excerpts  of  which  will  be  included  in  the 
next  chapter),  a  variety  of  different  terms  are  used  to  signify  system  attributes.  These 
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terms  are  "characteristic,"  "capability"  and  "parameter,"  and  they  are  usually  preceded 
by  the  term  "performance".  Intermingled  use  of  these  different  terms  tends  to 
complicate  system-related  discussion.  In  contrast  to  the  official  guidance,  this  report 
will  replace  these  various  terms  with  a  single,  overarching  term,  "attribute".  This 
consistency  should  facilitate  the  reader’s  understanding  of  the  subject  matter. 

C.  SYSTEM  EFFECTIVENESS  -  KEY  ISSUES 

1.  System  Effectiveness  -  A  Complex  Topic 

In  the  system  acquisition  process,  the  effectiveness  and  the  cost  of  a  system  are 
the  two  key  ingredients  in  determining  overall  system  value  to  the  military  services. 
Of  these  two  key  ingredients,  system  effectiveness  is  the  one  most  difficult  to  define, 
explain,  and  measure.  For  instance,  while  system  cost  may  be  defined  in  terms  of 
dollars,  system  effectiveness  has  no  such  standard  basis  for  definition.  Due  to  its 
inherent  complexity,  the  topic  of  system  effectiveness  has  received  wide  attention, 
and  debate,  throughout  the  DoD  in  recent  years.  In  the  current  era  of  DoD 
downsizing  and  reduced  budgets,  the  ability  to  relate  the  cost  of  a  system  to  an 
associated,  and  appropriate,  measure  of  effectiveness  will  become  increasingly 
important. 

2.  The  Key  Issues 

In  this  report,  the  author  will  attempt  to  add  to  the  system  effectiveness-related 
body  of  knowledge  by  addressing  the  following  key  issues: 

1 .  "What  is  system  effectiveness?" 

2.  "How  is  system  effectiveness  measured  or  quantified?" 

3.  "What  are  the  levels  at  which  system  effectiveness  should  be 
measured?" 

4.  "How  are  system  attributes  measured?" 
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5. 


"What  are  some  possible  system  effectiveness  models,  or  hierarchies, 
which  link  system  attribute  metrics  to  system  effectiveness  measure¬ 
ment?" 


As  a  prelude  to  later  examination  of  these  key  issues,  the  next  chapter  will 
present  applicable  excerpts  from  the  official  guidance  related  to  system  effectiveness 
in  the  system  acquisition  process. 
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III.  THE  OFFICIAL  GUIDANCE 


A.  INTRODUCTION 

This  chapter  will  present  the  applicable  guidance  related  to  the  topic  of  system 
effectiveness,  as  found  in  the  current  DoD  5000  series  instructions  and  the  U.S.  Code, 
Title  10.  Measures  of  effectiveness  (MOEs),  along  with  measures  of  performance 
(MOPs),  schedule,  and  cost,  are  developed  during  the  system  acquisition  process  and 
are  described  in  several  key  documents.  The  chapter  will  begin  with  a  brief  overview 
of  the  system  acquisition  process,  followed  by  a  detailed  description  of  the  pertinent 
information  included  in  the  key  documents.  The  Marine  Corps  Predator  Short  Range 
Assault  Weapon  (SRAW)  and  Advanced  Amphibious  Assault  Vehicle  (AAAV) 
systems  will  be  used  as  examples  throughout  the  chapter. 

B.  THE  SYSTEM  ACQUISITION  PROCESS  -  AN  OVERVIEW 

The  five  major  milestone  decision  points  and  the  five  phases  of  the  system 
acquisition  process,  illustrated  in  Figure  3.1,  provide  the  basis  for  comprehensive 
management  and  progressive  decisionmaking  associated  with  program  maturation. 
The  process  begins  with  the  development  of  the  Mission  Need  Statement  (MNS). 
Milestone  0,  Concept  Studies  Approval,  marks  the  initial  formal  interface  between  the 
system  requirements  generation  and  subsequent  system  acquisition  management 
processes.  Milestone  1  marks  the  official  start  of  a  new  system-related  acquisition 
program.  Subsequent  phases  and  milestone  decision  points  facilitate  the  orderly 
translation  of  broadly  stated  mission  needs  into  system  attribute  requirements  and  a 
stable  design  that  can  be  produced  efficiently. 
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Figure  3.1.  Acquisition  Milestones  &  Phases 

C.  THE  KEY  DOCUMENTS  AND  THEIR  RELATED  OUTPUTS 

The  key  documents  in  the  system  acquisition  process  are  the  Mission  Need 
Statement  (MNS),  the  Cost  and  Operational  Effectiveness  Analyses  (COEA),  the 
Operational  Requirements  Document  (ORD),  the  Test  and  Evaluation  Master  Plan 
(TEMP),  and  the  Acquisition  Program  Baseline  (APB).  Using  actual  examples  from 
the  Marine  Corps  Predator  SRAW  system,  Table  3.1  provides  an  overview  of  the 
primary  outputs  of  these  documents.  With  the  exception  of  the  MNS,  these 
documents  require  updating  through  the  system  acquisition  process  as  the  system 
evolves  from  a  concept  to  a  final  end  product. 
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DOCUMENT 

OUTPUT 

EXAMPLE  (PREDATOR) 

MNS 

Desired  Operational  Capabilities 

The  System  must  be  capable  of  being  fired  safely  from 
enclosed  areas  such  as  buildings  or  bunkers. 

COEA 

Functional  Objectives  (FOs) 

System  must  defeat  tanks  and  armored  vehicles  equipped 
with  explosive  reactive  armor. 

Measures  of  Effectiveness  (MOEs) 

Percent  Enemy  Armored  Fighting  Vehicles  Destroyed. 

Measures  of  Performance  (MOPs) 

Must  have  a  range  of  at  least  600m. 

ORD 

Performance  Parameters 

Warhead  must  be  capable  of  meeting  the  threshold 
target-defeat  criteria  against  a  Soviet  T-80  MBT  with 

ERA. 

Logistics  and  Readiness 

Requirements 

Predator  will  consist  of  a  launcher  and  missile  as  a 
singular  unit  for  issue.  The  launcher  will  be  considered  a 
disposable  item  upon  firing  of  the  missile. 

Critical  System  Characteristics 
(CSCs) 

The  system  must  be  capable  of  operating  properly  in  all 
battefield  environments  with  its  only  limitation  being  the 
capabilities  of  the  current  crew-served  night  vision  sight. 

TEMP 

Critical  Operational  Issues  (COIs) 

Will  the  system  enhance  the  chances  of  survivability  of 
Marines  firing  it  in  comparison  to  the  current  light 
antiarmor  system? 

Minimum  Acceptable  Operational 
Performance  Requirements 
(MAOPRs) 

System  will  be  man-portable,  not  to  exceed  20  lbs  in 
weight  and  40  inches  in  length. 

Critical  Technical  Parameters 
(CTPs) 

Accuracy  for  stationary  target  >  .5Ph  at  600m. 

APB 

Key  Performance  Parameters 
(KPPs) 

Stationary  Target  Ph  Threshold  .5  at  600m/Objective  .8  at 
600m. 

Table  3.1.  The  Key  Documents  and  Their  Primary  Outputs  [Ref.  3] 


D.  THE  MISSION  NEED  STATEMENT  (MNS) 

The  system  acquisition  process  begins  with  the  development  of  a  mission  need 
statement  (MNS)  by  the  user  or  operating  command.  To  assist  the  reader’s 
understanding  of  the  DoD  guidance  concerning  the  MNS,  definitions  and  examples 
of  "operational  scenario"  and  "mission"  are  given  below: 
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1.  Operational  Scenario 

Operational  scenarios  are  realistic  descriptions  of  potential  wartime  operations. 
An  example  of  an  operational  scenario,  from  the  AAAV  Supplemental  Analysis,  is 
provided: 


This  scenario  included  two  Marine  Expeditionary  Forces  (MEF),  one 
ashore  and  one  afloat.  In  a  coordinated  counteroffensive,  the  land- 
based  MEF  attacks  northwest,  while  the  afloat  MEF  lands  north  of  the 
Orange  (enemy)  forces.  The  concept  is  to  trap  and  neutralize  the 
enemy  units  between  the  two  MEFs.  Early  in  the  counterattack,  the 
following  preparatory  actions  were  taken: 

A  Blue  Regiment  conducted  a  helibome  assault  to  remove  an 
artillery  threat  to  two  Littoral  Penetration  Sites  (LPS)  selected 
for  the  surface  assault. 

A  Marine  Infantry  Regiment  and  an  Artillery  Battalion  were 
inserted  vertically  and  established  an  inland  fire  base  to  support 
the  surface  assault. 

A  Marine  Infantry  Regiment  landed  vertically,  west  of  the 
Littoral  Penetration  Zone  (LPZ)  and  established  blocking 
positions  to  facilitate  a  surface  assault  by  the  afloat  Regimental 
Landing  Team  (RLT). 


Continuing  with  the  scenario,  a  surface  amphibious  assault  begins 
approximately  one  hour  after  blocking  positions  were  established  by  the 
vertical  assault.  A  Marine  RLT  afloat,  which  consists  of  4,500 
Marines,  is  tasked  to  conduct  a  night  surface  assault  across  two 
designated  beaches  and  maneuver  to  rapidly  seize  an  inland  town.  The 
RLT  will  then  establish  blocking  positions  in  the  vicinity  of  the  town 
to  trap  Orange  forces  attempting  escape  to  the  northwest.  [Ref.  4:pp. 
iv-v] 
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2.  Mission  and  Mission  Tasks 

A  mission  is  an  operational  task  performed  by  military  personnel  and/or 
military  systems  as  part  of  an  operational  scenario.  Most  missions  can  be  sub-divided 
into  measurable  mission  tasks  or  functional  objectives.  For  example,  the  aforemen¬ 
tioned  RLT  mission  of  trapping  Orange  forces  consists  of  several  mission  tasks,  some 
of  which  relate  directly  to  the  AAAV,  such  as  the  ship-to-shore  movement  of  4,500 
Marines  across  two  designated  beaches  in  a  night  assault. 

3.  The  DoD  Guidance 

The  DoD  guidance  concerning  the  MNS  states: 


1.  Mission.  Identify  and  describe  the  mission  need  or 
deficiency.  Define  the  need  in  terms  of  mission,  objectives,  and 
general  capabilities.  Do  not  discuss  the  need  in  terms  of  specific 
equipment  or  system-specific  performance  characteristics. 

2.  Constraints.  Describe,  as  applicable,  key  boundary 
conditions  related  to  infrastructure  support  that  may  impact  on 
satisfying  the  need;  logistics  support;  transportation;  mapping;  charting 
and  geodosy  support;  manpower;  personnel;  and  training  constraints; 
command,  control,  communications,  and  intelligence  interfaces; 
security;  and  standardization  or  interoperability  within  the  North 
Atlantic  Treaty  Organization  (NATO)  or  with  other  allies  or  DoD 
Components.  Address  the  operational  environments  (including 
conventional;  initial  nuclear  weapons  effects;  nuclear,  biological,  and 
chemical  contamination  (NBCC);  electronic;  and  natural)  in  which  the 
mission  is  expected  to  be  accomplished.  Define  the  level  of  desired 
mission  capability  in  these  environments.  [Ref.  5:p.  2-1-1] 
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E.  THE  COST  AND  OPERATIONAL  EFFECTIVENESS  ANALYSIS 

(COEA) 

Based  on  the  MNS,  a  cost  and  operational  effectiveness  analysis  (COEA)  is 
developed  for  several  alternative  concepts.  The  COEA  is  the  most  important 
document  in  the  system  effectiveness  measurement  process.  As  part  of  the  COEA, 
mission  needs  are  sub-divided  into  functional  objectives  (mission  tasks).  Also, 
alternative  system  concepts  which  fulfill  the  desired  mission  need  are  compared  using 
Measures  of  Effectiveness  (MOEs)  and  Measures  of  Performance  (MOPs).  The  DoD 
guidance  concerning  the  COEA  states: 


1.  COEAs  aid  decision  making  by  illuminating  the  relative 
advantages  and  disadvantages  of  the  alternatives  being 
considered  and  showing  the  sensitivity  of  each  alternatives  to 
possible  changes  in  key  assumptions  (e.g.,  the  threat)  or 
variables  (e.g.,  selected  performance  capabilities).  Accordingly, 
the  analysis  takes  the  form  of  a  problem  of  choice.  The  COEA 
should  aid  decisionmakers  in  judging  whether  or  not  any  of  the 
proposed  alternatives  to  the  current  program  (i.e.,  the  status  quo) 
offer  sufficient  military  benefit  to  be  worth  the  cost. 

2.  The  COEA  will  typically  draw  on  several  sub-analyses.  These 
include  analyses  of  mission  needs,  the  threat  and  U.S. 
capabilities,  the  interrelationship  of  systems,  the  contribution  of 
multi-role  systems,  measures  of  effectiveness,  costs,  and  cost- 
effectiveness  comparisons. 

3.  Functional  Objectives  (FOs).  Statements  describing,  in 
quantitative  terms,  the  tasks  a  system  needs  to  perform  in 
accomplishing  the  mission(s).  They  depend  on  the  type  of 
system  at  issue.  For  example,  when  analyzing  transportation 
systems,  functional  objectives  are  stated  in  movement 
requirements.  For  firepower  systems,  they  reflect  the  types  of 
targets  to  be  engaged.  The  effectiveness  of  system  alternatives 
is  the  direct  result  of  the  degree  to  which  the  functional 
objectives  are  attained.  [Ref.  5:pp.  8-4  and  8-5] 
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4.  Measures  of  Effectiveness  (MOEs).  To  judge  whether  an 
alternative  is  worthwhile,  one  must  first  determine  what  it  takes 
to  make  a  difference.  Measures  of  effectiveness  should  be 
defined  to  measure  operational  capabilities  in  terms  of 
engagement  or  battle  outcomes. 

5.  Measures  of  Performance  (MOPs).  Measures  of  performance, 
such  as  weight  and  speed,  should  relate  to  the  measures  of 
effectiveness  such  that  the  effect  of  a  change  in  the  measure  of 
performance  can  be  related  to  a  change  in  the  measure  of 
effectiveness.  [Ref.  6:pp.  4-E-l  -4-E-3] 

F.  THE  OPERATIONAL  REQUIREMENTS  DOCUMENT  (ORD) 

Based  on  the  initial  COEA,  an  Operational  Requirements  Document  (ORD) 
is  developed.  The  ORD  translates  the  COEA  FOs/MOEs/MOPs  into  system-specific 
operational  requirements.  Thus,  the  ORD  defines  the  system  attribute  requirements, 
or  performance  parameters  necessary  to  bring  about  the  overall  system  effectiveness 
defined  in  the  COEA.  To  assist  the  reader’s  understanding  of  the  DoD  guidance 
concerning  the  ORD,  the  DoD  definition  of  performance  parameter  is  provided: 

1.  Performance  Parameter 

Those  operational  and  support  characteristics  of  a  specific  system  that 
allow  it  to  effectively  and  efficiently  perform  its  assigned  mission  over 
time.  The  support  characteristics  of  the  system  include  both  support- 
ability  aspects  of  design  and  the  support  elements  necessary  for  system 
operation.  [Ref.  6:p.  15-13] 

2.  The  DoD  Guidance 

The  DoD  guidance  concerning  the  format  for  the  ORD  states: 
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a.  Capabilities  Required 

Identify  performance  capabilities  and  characteristics  required. 
State  in  operational  terms  and  prioritize  if  possible.  Specify  each 
performance  parameter  in  terms  of  a  minimum  acceptable  value 
(threshold)  required  to  satisfy  the  mission  need  and  a  performance 
objective.  The  objective  should  represent  a  measurable,  beneficial 
increase  in  capability  or  operations  and  support  above  the  threshold. 

(1)  System  Performance.  Include  system  perform¬ 
ance  parameters  such  as  range,  accuracy,  payload,  speed,  mission 
reliability,  etc.  Describe  mission  scenarios  (wartime  and  peacetime,  if 
different  in  terms  of  mission  profiles,  employment  tactics,  and 
environmental  conditions  (all  inclusive;  natural  and  man-made,  e.g., 
weather,  countermeasures,  ocean  acoustics,  etc). 

(2)  Logistics  and  Readiness.  (These  are  measures 
related  to  system  supportability).  Include  measures  of  mission-capable 
rate,  operational  availability,  frequency  and  duration  of  preventive  or 
scheduled  maintenance  actions,  etc.  Describe  in  terms  of  mission 
requirements  considering  both  wartime  and  peacetime  logistics 
operations.  Identify  combat  support  require-ments  including  battle 
assessment  damage  repair  capability,  mobility  requirements,  expected 
maintenance  manpower  and  skill  levels,  and  surge  and  mobilization 
objectives  and  capabilities.  [Ref.  5:pp.  3-1-1  -  3-1-2] 

(3)  Critical  System  Characteristics.  (These  are)  those 
design  features  that  determine  how  well  the  proposed  concept  or  system 
will  function  in  its  intended  environment.  Critical  system  characteris¬ 
tics  include  survivability,  transportability,  energy  efficiency,  and 
interoperability,  standardization,  and  compatibility  with  other  forces 
and  systems  including  support  infrastructure.  [Ref.  6:p.  4-C-l] 

Address  electronic  counter  countermeasures  (ECCM)  and  Wartime 
Reserve  Modes  (WARM)  requirements;  conventional,  initial  nuclear 
weapons  effects,  and  nuclear,  biological,  and  chemical  contamination 
(NBCC),  survivability;  natural  environmental  factors  (such  as  climatic, 
terrain,  and  oceanographic  factors);  and  electromagnetic  compatibility 
and  frequency  spectrum  assignment  for  systems  operating  in  the 
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electromagnetic  spectrum.  Define  the  expected  mission  capability 
(e.g.,  full,  percent  degraded,  etc)  in  the  various  environments.  Include 
applicable  safety  parameters  such  as  those  related  to  system,  nuclear, 
explosive,  and  flight  safety.  Identify  communications,  information,  and 
physical  and  operational  security  needs.  [Ref.  5:p.  3-1-2] 

G.  THE  TEST  AND  EVALUATION  MASTER  PLAN  (TEMP) 

The  TEMP  identifies  the  necessary  operational  test  and  evaluation  (OT&E) 
activities  and  translates  the  COEA  and  ORD  effectiveness  and  performance 
requirements  into  testable  parameters.  To  assist  the  reader’s  understanding  of  the 
DoD  guidance  concerning  the  TEMP,  definitions  for  "operational  effectiveness"  and 
"operational  suitability",  along  with  the  U.S.  Code,  Title  10  guidance  concerning 
operational  testing,  are  provided: 


1.  Operational  Effectiveness.  The  overall  degree  of  mission 
accomplishment  of  a  system  when  used  by  representative 
personnel  in  the  environment  planned  or  expected  (e.g.,  natural, 
electronic,  threat  etc.)  for  operational  employment  of  the  system 
considering  organization,  doctrine,  tactics,  survivability, 
vulnerability,  and  threat  (including  countermeasures,  initial 
nuclear  weapons  effects,  nuclear,  biological,  and  chemical 
contamination  (NBCC)  threats). 

2.  Operational  Suitability.  The  degree  to  which  a  system  can  be 
placed  satisfactorily  in  field  use  with  consideration  given  to 
availability,  compatibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintainability,  safety,  human 
factors,  manpower  supportability,  logistic  supportability,  natural 
environmental  effects,  documentation,  and  training.  [Ref.  6:p. 
15-13] 

3.  U.S.  Code,  Title  10  Guidance.  (The  term  "operational  test  and 
evaluation"  means)  the  field  test,  under  realistic  combat 
conditions,  of  any  item  of  (or  key  component  of)  weapons, 
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equipment,  or  munitions  for  the  purpose  of  determining  the 
effectiveness  and  suitability  of  the  weapons,  equipment,  or 
munitions  for  use  in  combat  by  typical  military  users.  [Ref.  7:p. 
400] 

4.  The  DoD  Guidance.  The  DoD  guidance  concerning  the  TEMP 
format  states: 

a.  Critical  Operational  Issues 

(1)  List  in  this  section  the  critical  operational  issues. 
Critical  operational  issues  are  the  operational  effectiveness  and 
operational  suitability  issues  (not  parameters,  objectives,  or  thresholds) 
that  must  be  examined  in  operational  test  and  evaluation  to 
evaluate/assess  the  system's  capability  to  perform  its  mission.... 

(2)  If  every  critical  operational  issue  is  resolved 
favorably,  the  system  should  be  operationally  effective  and  opera¬ 
tionally  suitable  when  employed  in  its  intended  environment  by  typical 
users.  [Ref.  5 :pp.  7-1-2  and  7-1-3] 

(a)  Minimum  Acceptable  Operational  Per¬ 
formance  Requirements.  Reference  the 

Operational  Requirements  Document  and  summarize  the  critical 
operational  effectiveness  and  suitability  parameters  and  constraints 
(manpower,  personnel,  training,  software,  computer  resources,  trans¬ 
portation  (lift),  and  etc.)  described  therein. 

(b)  Critical  Technical  Parameters.  List  in 
matrix  format  the  critical  technical  parameters  of  the  system  (including 
software  maturity  and  performance  measures)  that  have  been  evaluated 
or  will  be  evaluated  during  the  remaining  phases  of  developmental 
testing.  Critical  technical  parameters  are  derived  from  the  Operational 
Requirements  Document,  critical  system  characteristics,  and  technical 
performance  measures  and  should  include  the  parameters  in  the 
acquisition  program  baseline. 

Next  to  each  technical  parameter,  list  the  accompanying 
objectives  and  thresholds...  [Ref.  5:pp.  7-1-3] 
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H.  THE  ACQUISITION  PROGRAM  BASELINE  (APB) 

The  APB  is  the  program  manager’s  tool  for  monitoring  system  development 
and  for  identifying  key  cost,  schedule,  and  performance  parameters  for  Milestone 
decision  making.  Key  performance  parameters  are  found  in  the  Performance  Baseline 
of  the  APB.  To  assist  the  reader’s  understanding  of  the  DoD  guidance  concerning  the 
performance  portion  of  the  APB,  definitions  for  "key  performance  parameters", 
thresholds  ,  and  objectives",  and  an  example  of  key  performance  parameters  for  a 
particular  system,  the  AAAV(F),  are  provided: 

1.  Key  Performance  Parameters.  "Those  system-related  parameters  that 
if  the  thresholds  are  not  met,  the  milestone  decision  authority  would 
require  a  reevaluation  of  alternative  concepts  or  design  approaches" 
[Ref.  6:p.  11-A-l-l] 

2.  Thresholds.  "Thresholds  are  the  minimum  acceptable  value  required 
to  satisfy  the  mission  need."  [Ref.  6:p.  4-B-l  ] 

3.  Objectives.  Objectives  are  also  provided  for  key  parameters,  and  are 
defined  as  a  "measurable,  beneficial  increase  in  capability  or  operations 
support  above  the  threshold."  [Ref.  5:p.  3-1-1] 

4.  Key  Performance  Parameters  -  Example. 

Table  3.2  shows  the  key  performance  parameters  from  the  AAAV(F) 
Acquisition  Program  Baseline,  with  related  system  attributes  provided  by  the  author. 
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KEY  PERFORMANCE 
PARAMETER 

OBJECTIVE/THRESHOLD 

RELATED  SYSTEM 
ATTRIBUTE 

High  Water  Speed  (knots) 

25/20 

Speed 

Forward  Speed  on  a  Hard 

Surface  Road  (k/h) 

72/69 

Speed 

Carrying  Capacity  (Marines) 

18/17 

Payload 

Firepower  (Maximum 

Effective  Range) 

2000/1500 

Lethality 

Armor  Protection  Against 
(millimeter/meter) 

(30/1 000)/(  1 4.5/300) 

Survivability 

Mean  Time  Between  Critical 
Mission  Failure  (hrs) 

(MTBCMF) 

95/70 

Reliability 

Table  3.2.  AAAV(F)  Key  Performance  Parameters  [Ref.  8] 


5.  The  DoD  guidance.  The  DoD  guidance  concerning  the  performance 

baseline  portion  of  APB  states: 

a.  The  Performance  Baseline 

Each  commodity  has  a  few  parameters  which  are  critical  to  that 
commodity  and  must  be  addressed  (e.g.,  aircraft  weight,  missile  range, 
reliability).  List  these  key  critical  parameters....  Enter  acquisition 
program  baseline  performance  requirements  for  parameters  tailored  to 
each  program.  Performance  objectives  and  thresholds  will  be  derived 
from  the  Operational  Requirements  Document  and  the  results  of  the 
previous  acquisition  phase.  Performance  objectives  and  thresholds 
must  be  reviewed  by  the  Joint  Requirements  Oversight  Council  (for 
acquisition  category  I D  programs)  at  each  milestone,  and  ultimately  be 
verifiable  by  developmental  and  operational  testing.  Performance 
includes  operational,  technical,  and  supportability  parameters.  [Ref. 

5:pp.  14-1-3  and  14-1-7] 
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I.  THE  NEW  DRAFT  DODI  5000.2  GUIDANCE 

As  of  October  1995,  an  updated  revision  of  the  DoDI  5000.2  has  been  issued 
for  review  by  the  DoD  community.  In  this  revision,  which  has  not  yet  been  formally 
approved,  the  guidance  concerning  the  aforementioned  key  documents  does  not 
significantly  change,  except  for  the  guidance  concerning  the  COEA.  The  new  draft 
DoDI  5000.2  replaces  the  current  term  "COEA"  with  the  term  "Analysis  of 
Alternatives",  along  with  the  following,  significantly  less  specific  guidance: 


...These  analyses  are  intended  to  aid  and  Document  Decisionmaking  by 
illuminating  the  relative  advantages  and  disadvantages  of  the 
alternatives  being  considered.  Show  the  sensitivity  of  each  alternative 
to  possible  changes  in  key  assumptions  (e.g.,  threat)  or  variables  (e.g., 
selected  performance  capabilities).  The  analysis  shall  aid  decision¬ 
makers  in  judging  whether  or  not  any  of  the  proposed  alternatives  to  an 
existing  system  offer  sufficient  military  benefit  to  be  worth  the  cost. 
There  shall  be  a  clear  linkage  between  the  analysis  of  alternatives,  and 
requirements,  and  system  evaluation  measures  of  effectiveness.  [Ref. 
9:p.  2.2] 


The  reader  should  be  aware  that  this  change,  if  ultimately  accepted,  may  affect 
some  of  the  information  in  this  thesis.  However,  since  the  updated  guidance  has  not 
been  formally  approved,  the  author  will  base  this  thesis  on  the  existing  DoDI  5000.2 
guidance. 

J.  SUMMARY  OF  OFFICIAL  GUIDANCE 

The  MNS,  COEA,  ORD,  TEMP,  and  APB  are  the  key  documents  used  in 
developing  systems  in  response  to  mission  needs.  It  is  important  to  note  that  the 
COEA,  ORD,  TEMP,  and  APB  are  not  static  documents,  but  are  updated  throughout 
the  system  development  process  as  the  system  is  developed  in  greater  detail.  The 


21 


provided  excerpts  from  the  official  guidance  establish  a  basis  for  further  examination 
of  system  effectiveness  measurement  and  its  relation  to  measures  of  system  attributes. 

K.  BEYOND  THE  OFFICIAL  GUIDANCE 

Despite  the  detailed  information  provided  in  this  chapter,  there  are  certain 
aspects  of  system  effectiveness  which  require  further  clarification.  Specifically,  the 
official  guidance  inadequately  addresses  the  levels  at  which  system  effectiveness  can 
be  measured,  the  definition  of  system  effectiveness,  the  measurement  of  system 
attributes,  and  possible  models/hierarchies  for  linking  measures  of  system  attributes 
to  measures  of  system  effectiveness. 

The  remainder  of  this  thesis  will  provide  information  related  to  these  missing 
aspects  of  system  effectiveness  guidance.  In  the  next  chapter,  the  author  will  offer  a 
alternative  viewpoint  to  the  official  guidance  concerning  the  level(s)  at  which  system 
effectiveness  should  be  measured,  which  will  then  be  incorporated  into  a  definition 
for  system  effectiveness.  In  the  remaining  chapters,  some  various  models/hierarchies 
for  linking  measures  of  system  attributes  to  measures  of  system  effectiveness  will  be 
offered. 
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IV.  SYSTEM  EFFECTIVENESS  -  TWO  LEVELS 


A.  INTRODUCTION 

In  the  previous  chapter,  the  current  official  guidance  concerning  system 
effectiveness  stated  that  measures  of  effectiveness  (MOEs)  must  be  made  in  terms  of 
engagement  or  battle  outcome  (e.g.,  loss-exchange  ratios,  residual  force  ratios,  etc.). 
In  this  chapter,  the  author  will  provide  an  additional  viewpoint,  that  system 
effectiveness  should  also  be  measured  in  terms  of  mission  accomplishment  (e.g., 
capability  to  accomplish  logistic  resupply  operations,  capability  to  accomplish  TRAP 
operations).  This  additional  viewpoint  will  be  shown  to  be  consistent  with  1)  the 
DoD  guidance  concerning  the  Concept  Exploration  and  Definition  (CE/D)  Phase,  2) 
the  Operational  Test  and  Evaluation  (OT&E)  process,  and  3)  Army  and  Marine  Corps 
guidance.  Furthermore,  excerpts  from  the  Medium  Lift  Replacement  (MLR)  study, 
in  which  the  MV-22  was  chosen  over  alternative  systems,  will  show  that  mission- 
level  MOEs  are  already  in  use.  Finally,  an  argument  will  be  made  that  measuring 
system  effectiveness  at  the  mission  level,  in  addition  to  the  battle  level,  provides  a 
better  opportunity  for  system  development  and  test  personnel  to  link  system  attributes 
to  system  effectiveness. 

B.  THE  OFFICIAL  GUIDANCE  -  MEASURES  OF  EFFECTIVENESS 

The  official  guidance  concerning  measures  of  effectiveness,  given  in  Chapter 
III  and  shown  below,  specifies  only  one  level  at  which  system  effectiveness  should 
be  measured. 

To  judge  whether  or  not  an  alternative  is  worthwhile,  one  must  first 
determine  what  it  takes  to  make  a  difference.  Measures  of 
effectiveness  should  be  defined  to  measure  operational  capabilities 
in  terms  of  engagement  or  battle  outcomes.  [Ref.  6:p.  4-E-3] 
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The  following  sections  will  demonstrate  that  MOEs  should  also  be  defined  in 
terms  of  mission  outcomes. 

C.  THE  CONCEPT  EXPLORATION  AND  DEFINITION  (CE/D)  PHASE 
1.  The  Official  Guidance 

In  contrast  to  the  current  battle-level  MOE  guidance,  other  guidance 
concerning  the  CE/D  Phase  leads  to  an  additional  viewpoint  as  to  the  level  at  which 
system  effectiveness  can  be  measured.  The  following  excerpts,  which  were  excluded 
from  the  previous  chapter  due  to  their  inapplicability  to  a  particular  key  document, 
support  this  argument: 


a.  Milestone  0,  Concept  Studies  Approval,  marks  the  initial 
formal  interface  between  the  requirements  generation 
and  acquisition  management  systems.  As  a  result  of  this 
review  (in  the  CE/D  Phase),  studies  are  conducted  of 
materiel  concepts  to  identify  the  most  promising 
potential  solution(s)  to  validated  mission  needs. 

b.  Subsequent  phases  and  milestone  decision  points 
facilitate  the  orderly  translation  of  broadly  stated 
mission  needs  into  system  specific  performance 
requirements  and  a  stable  design  that  can  be  produced 
efficiently.  [Ref.  6:  pp.  2-1  and  2-2] 


2.  Two  Levels  for  Measurement 

a.  Two  Levels  for  Measuring  System  Effectiveness 

From  the  MOE  guidance  and  CE/D  phase  guidance,  one  can  arrive  at 
two  possible  levels  for  measuring  system  effectiveness.  On  the  one  hand,  the  MOE 
guidance,  "measures  of  effectiveness  should  be  defined  to  measure  operational 
capabilities  in  terms  of  engagement  or  battle  outcomes",  leads  to  the  conclusion  that 
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system  effectiveness  must  be  measured  at  the  battle  level.  On  the  other  hand,  the 
CE/D  phase  guidance,  "studies  are  conducted  of  materiel  concepts  to  identify  the  most 
promising  potential  solution(s)  to  validated  mission  needs",  and  "translation  of 
broadly  stated  mission  needs",  leads  to  the  conclusion  that  system  effectiveness 
should  also  be  measured  in  terms  of  mission  accomplishment,  at  the  mission  level. 

b.  Mission  Needs 

Now,  there  would  be  no  inconsistency  between  these  two  levels  if 
mission  needs  were  stated  explicitly  in  terms  of  desired  engagement  or  battle 
outcomes.  However,  this  is  usually  not  the  case.  The  following  excerpts  from  the 
official  guidance  concerning  the  mission  need  statement  (MNS)  support  this 
argument: 


(1)  Mission.  Identify  and  describe  the  mission  need 
or  deficiency.  Define  the  need  in  terms  of  mission,  objectives,  and 
general  capabilities.  Do  not  discuss  the  need  in  terms  of  specific 
equipment  or  system-specific  performance  characteristics. 

(2)  Constraints.  Describe,  as  applicable,  key 
boundary  conditions  related  to  infrastructure  support  that  may  impact 
on  satisfying  the  need;  logistics  support;  transportation;  mapping; 
charting  and  geodosy  support;  manpower;  personnel;  and  training 
constraints;  command,  control,  communications,  and  intelligence 
interfaces;  security;  and  standardization  or  interoperability  within  the 
North  Atlantic  Treaty  Organization  (NATO)  or  with  other  allies  or 
DoD  Components.  Address  the  operational  environments  (including 
conventional;  initial  nuclear  weapons  effects;  nuclear,  biological,  and 
chemical  contamina-tion  (NBCC);  electronic;  and  natural)  in  which  the 
mission  is  expected  to  be  accomplished.  Define  the  level  of  desired 
mission  capability  in  these  environments.  [Ref.  5  :p.  2-1-1) 
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3.  Summary 

Since  mission  need  statements  are  expressed  in  terms  of  operational 
capabilities  required  to  conduct  missions  (at  the  mission  level),  the  search  for  system 
solutions  to  those  needs  requires  measuring  the  effectiveness  of  alternative  systems 
at  both  the  battle  and  mission  level.  For  some  major  systems,  such  as  an  aircraft 
carrier,  mission  accomplishment  might  best  be  stated  as  an  engagement  or  battle 
outcome.  However,  for  most  systems,  which  operate  at  a  lower  level,  mission 
accomplishment  is  an  essential  intermediate  step  that  combines  with  a  multitude  of 
other  factors  to  ultimately  result  in  an  engagement  or  battle  outcome. 

D.  THE  OPERATIONAL  TEST  AND  EVALUATION  (OT&E)  PROCESS 

1.  The  DoD  Guidance 

Measuring  system  effectiveness  at  the  mission  level  is  also  consistent  with  the 
guidance  concerning  the  Operational  Test  and  Evaluation  process.  The  following 
excerpts  from  the  official  guidance  support  this  argument: 


The  term  "operational  test  and  evaluation"  means  "the  field  test,  under 
realistic  combat  conditions,  of  any  item  of  (or  key  component  of) 
weapons,  equipment,  or  munitions  for  the  purpose  of  determining  the 
effectiveness  and  suitability  of  the  weapons,  equipment,  or  munitions 
for  use  in  combat  by  typical  military  users.  [Ref.  7:p.  400] 

Operational  Effectiveness.  The  overall  degree  of  mission 
accomplishment  of  a  system  when  used  by  representative  personnel  in 
the  environment  planned  or  expected  (e.g.,  natural,  electronic,  threat 
etc.)  for  operational  employment  of  the  system  considering  organiza¬ 
tion,  doctrine,  tactics,  survivability,  vulnerability,  and  threat  (including 
countermeasures,  initial  nuclear  weapons  effects,  nuclear,  biological, 
and  chemical  contamination  (NBCC)  threats).  [Ref.  6:p.  15-13] 


26 


2. 


Two  Levels 


As  in  the  earlier  section,  one  could  argue  that  the  level  for  measuring 
operational  effectiveness  (mission  accomplishment)  is  usually  different  than  the 
currently-mandated  level  for  measures  of  effectiveness  (engagements  or  battle 
outcomes). 

3.  The  Commander,  OT&E  Draft  "Battle  Outcome"  MOEs  Letter 

The  following  excerpt  from  a  draft  letter  dated  22  February  1994,  from  the 
Commander  of  the  Navy’s  operational  test  and  evaluation  force,  highlights  some 
serious  problems  related  to  the  current  guidance  concerning  the  battle  level  for 
MOEs: 


OSD/DOT&E  are  interpreting  DODINST  5000.2  requirements  for 
linkage  of  COEA  MOEs  to  operational  testing  and  the  use  of  battle 
outcome  MOEs  with  such  rigidity  that  I  believe  this  causes  a  significant 
threat  to  all  Navy  oversight  programs.  The  problem  with  this  approach 
is  that  a  requirement  to  use  high-level  MOEs,  for  many  systems, 
obscures  real  effectiveness  and  suitability  defining  measures.  This 
could  lead  directly  to  poor  acquisition  decisions. 

The  current  OSD/DOT&E  philosophy  is  that  all  systems  can  and  must 
be  tested  in  terms  of  "outcome  of  the  battle"  MOEs,  and  that  all  these 
MOEs  must  be  thresholded.  I  do  not  believe  this  is  a  sound  policy  for 
now  or  in  the  future.  Most  of  the  systems  we  test  will  not  determine  the 
outcome  of  a  battle  and  they  are  only  one  contributor  among  a  number 
of  powerful  factors  which  will  determine  the  outcome  of  an 
engagement.... 

The  less  influence  a  system  exerts  on  the  outcome  of  the  battle,  the 
more  obscure  the  effectiveness  and  suitability  of  these  systems  in  high- 
level  MOEs  become....  The  current  trend  of  overemphasis  on  battle 
outcome  level  MOEs  and  the  strict  requirement  for  their  linkage  will 
directly  lead  to  unsatisfactory  results;  important  acquisition  decisions 
will  be  based  on  inappropriate  parameters;  and  the  cost  of  testing  will 
increase  because  battle  outcome  MOEs  will  require  considerably  more 
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testing  than  necessary  to  isolate  the  system’s  true  effectiveness....  I 
believe  we  should  not  transfer  this  problem  to  the  future.  Now  is  the 
time  for  a  solution....  The  goal  for  the  long-term  and  short-term 
approach  is  to  ensure  testable,  measurable,  and  meaningful  MOEs  are 
used  to  evaluate  a  weapon  system’s  worth.  [Ref.  10:pp.  1-2] 

4.  Summary 

The  operational  test  community  would  benefit  from  allowing  the  measurement 
of  system  effectiveness  at  both  the  battle  and  mission  levels.  One  can  derive  from  the 
Commander,  OT&E  letter  that,  for  many  systems  which  do  not  contribute  directly  to 
a  battle-level  outcome,  mission-level  MOEs  may  provide  a  better  indication  of  a 
system’s  true  effectiveness  and  suitability. 

E.  ARMY  AND  MARINE  CORPS  GUIDANCE 
1.  The  Guidance 

A  primary  source  of  guidance  used  by  the  Army  and  the  Marine  Corps  for 
measuring  system  effectiveness  is  the  TRADOC  PAM  1 1-8,  "Operational  Effective¬ 
ness".  The  following  excerpts  support  the  measurement  of  system  effectiveness  at  the 
mission  level. 


Effectiveness  is  the  degree  (or  measure)  to  which  a  system  performs  its 
stated  mission.  The  system  is  effective  if  it  does  what  the  require¬ 
ment  specifies....  An  MOE  is  a  quantitative  indicator  of  the  ability  of 
a  system  to  accomplish  its  designated  task.  [Ref.  1  l:p.  12] 


2.  Summary 

Thus,  the  argument  for  measuring  system  effectiveness  at  the  mission  level  is 
enhanced  by  the  TRADOC  guidance.  In  reference  to  system  effectiveness,  the 
publication  uses  the  terms  "stated  mission",  "the  requirement",  and  "designated  task". 
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These  are  obviously  not  references  to  some  battle  outcome,  but  are  rather  references 
to  specific  mission  accomplishment. 

F.  THE  MEDIUM  LIFT  REPLACEMENT  (MLR)  STUDY 

1.  Background 

In  the  Medium  Lift  Replacement  (MLR)  study,  or  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  [Ref.  12],  dated  24  August  1994,  the  MLR  aircraft 
included  the  MV-22  Osprey  and  various  helicopters.  Using  the  terms  "operational" 
and  "tactical",  the  MLR  study  compared  the  alternative  aircraft  at  the  battle  and 
mission  levels.  Applicable  excerpts  from  the  study  are  given  below. 

2.  The  Operational  Level 

Table  4.1  shows  the  primary  level  MOEs  at  the  operational  (battle)  level. 


MOE  1-1 

Contributions  of  the  MLR  alternatives  to  "WIN  QUICKLY" 

MOE  1-2 

Contributions  of  the  MLR  alternatives  to  "WIN  DECISIVELY" 

MOE  1-3 

Contributions  of  the  MLR  alternatives  to  "DOMINATE 
BATTLESPACE" 

MOE  1-4 

Contributions  of  the  MLR  alternatives  to  "MINIMIZE 

CASUALTIES" 

MOE  1-5 

Contributions  of  the  MLR  alternatives  to  "GENERATE 
OVERWHELMING  TEMPO" 

Table  4.1.  MLR  Study  -  Primary  Operational  (Battle)  Level  MOEs 

At  this  level,  the  contributions  of  the  MLR  aircraft  to  joint  operations  and  the 
power  projection  capabilities  of  Naval  Expeditionary  Forces  (NEF)  were  evaluated. 
The  focus  was  on  the  effects  the  fleets  of  different  MLR  aircraft  have  on  the  battle 
outcomes  of  the  Blue  and  Orange  forces.  The  MOEs  were  stated  in  the  tenets  of 
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Operational  Maneuver  From  the  Sea  (OMFTS).  The  battle  was  represented  by  two 
operational  situations  (OPSITS)  modeled  in  the  Corps  Battle  Analyzer  (CORBAN). 
The  distinguishing  input  was  the  different  mission  completion  times  that  the  different 
MLR  aircraft  fleets  had  on  force  buildup  during  assault  operations.  In  addition  to  the 
primary  level  MOEs,  there  were  23  other  secondary  level  MOEs  such  as  orange 
battalions  neutralized,  loss-exchange  ratios,  etc. 

3.  The  Tactical  Level 

Table  4.2  shows  the  primary  MOEs  at  the  tactical  (mission)  level. 


MOE  2-1 

Capability  of  MLR  alternatives  to  build  up  forces 

MOE  2-2 

Survivability  of  MLR  alternatives 

MOE  3-1 

Capability  of  MLR  alternatives  to  accomplish  logistics  resupply  operations 

MOE  3-2 

Capability  of  MLR  alternatives  to  accomplish  MEDEVAC  operations 

MOE  3-3 

Capability  of  MLR  alternatives  to  accomplish  TRAP  operations 

MOE  3-4 

Capability  of  MLR  alternatives  to  accomplish  SAR  operations 

Table  4.2.  MLR  Study  Primary  Tactical  (Mission)  Level  MOEs 


On  the  tactical  level,  the  effects  of  the  MLR  alternatives  on  a  variety  of 
missions  were  evaluated.  At  this  level,  the  MOEs  focused  on  the  effects  the  fleets  of 
different  MLR  aircraft  had  on  force-buildup  rates,  logistics  resupply  operations, 
medevac  operations,  TRAP  operations,  and  SAR  operations.  Additionally,  an  MOE 
specifically  related  to  survivability  was  used.  In  the  evaluation  of  force  buildup  rates 
and  logistics  resupply  missions,  three  tactical  situations  (TACSITs)  modeled  in  the 
Amphibious  Assault  Model  (AAM)  were  used.  The  primary  inputs  for  the  AAM 
were  system  attributes  such  as  aircraft  speed,  altitude,  range,  payload,  fuel 
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expenditure,  reliability,  maintainability,  and  availability.  For  the  remaining  MOEs, 
spreadsheet  models  were  used,  all  with  system  attributes  as  inputs.  In  addition  to  the 
primary  level  MOEs,  there  were  1 1  other  secondary  level  MOEs  such  as  TRAP 
completion  time,  medevac  lives  saved,  etc. 

4.  Summary 

One  can  see  that  system  effectiveness  for  the  MLR  study  was  measured  at  both 
the  battle  (operational)  and  mission  (tactical)  levels.  At  the  battle  level,  the  CORBAN 
model  used  one  primary  input,  mission  completion  time,  to  produce  28  operational 
level  MOEs.  These  MOEs  were  used  to  show  the  degree  of  effectiveness  that  the  V- 
22,  the  system  with  the  shortest  mission  completion  time,  had  over  the  other  systems 
at  the  battle  level.  At  the  mission  level,  numerous  system  attributes  were  used  as 
inputs  to  the  various  models.  Seventeen  MOEs  were  used  to  show  the  degree  of 
effectiveness  that  the  V-22,  due  to  its  superior  performance  capabilities,  had  over 
competing  systems  at  the  mission  level. 

G.  SYSTEM  DESIGN  AND  TEST 

In  a  further  argument  for  mission-level  MOEs,  system  engineers  do  not  usually 
develop  systems  with  an  engagement  or  battle  outcome  in  mind.  They  develop 
systems  to  meet  specific  mission-related  requirements.  Thus,  an  effectiveness 
threshold  for  time  to  move  Marines  from  ship-to-shore  gives  the  engineer  a  basis 
upon  which  to  develop  requirements  for  system  attributes  such  as  speed  and  carrying 
capacity.  Likewise,  test  and  evaluation  personnel  do  not  usually  test  systems  to  meet 
certain  engagement  or  battle  outcome  objectives.  They  test  systems  to  see  if  desired 
system  attribute  requirements  are  met  and  if  the  system  meets  specific  mission 
requirements.  Although  battle-level  MOEs  are  useful  in  determining  how  systems 
will  affect  the  outcome  of  the  battle,  mission-level  MOEs  provide  a  better  basis  upon 
which  to  design  and  test  systems. 
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H.  AN  IMPORTANT  CONCLUSION 

All  of  the  above  analyses  lead  to  an  important  conclusion.  System 
effectiveness  should  be  measured  at  the  mission  level,  in  terms  of  mission  tasks 
(i.e.,  broad  operational  capabilities),  in  addition  to  the  battle  level,  in  terms  of 
engagement  or  battle  outcomes.  For  instance,  one  measure  of  effectiveness  for 
comparing  the  V-22  to  alternative  systems  could  be  stated  in  terms  of  mission 
completion  time  to  move  a  MEF-sized  assault  element  from  ship-to-shore. 
Obviously,  the  system  which  accomplishes  this  mission  in  the  shortest  amount  of 
time,  without  an  unacceptable  number  of  casualties,  will  be  the  more  effective  system 
in  meeting  that  particular  mission  need.  Stating  this  effectiveness  in  battle-related 
terms  may  or  may  not  be  reasonable,  depending  on  the  relation  of  the  system’s 
mission  to  battle-level  outcomes.  As  shown  in  the  MLR  study,  battle-level 
effectiveness  measures  for  major  systems  like  the  MV-22  may  be  derived  using 
models  based  on  mission-level  effectiveness  measures,  such  as  mission  completion 
time. 
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V.  THE  SYSTEM  DEVELOPMENT  PROCESS 


A.  INTRODUCTION 

The  military  services  develop  and  employ  systems  in  order  to  accomplish  their 
assigned  missions.  In  the  system  development  process,  the  measurement  of  system 
effectiveness  is  used,  along  with  the  associated  cost,  to  choose  between  competing 
system  designs.  Thus,  the  measurement  of  system  effectiveness  is  a  key  element  of 
the  system  acquisition  process. 

Previous  chapters  described  the  current  system  effectiveness  and  performance 
(attribute)  measurement  process,  and  established  that  system  effectiveness  should  be 
measured  at  both  the  battle  and  mission  levels.  This  chapter  will  provide  a  definition 
for  system  effectiveness  and  a  deeper  review  of  the  concept  of  system  effectiveness, 
with  emphasis  on  the  relationship  between  system  attributes,  system  design,  and 
system  effectiveness  at  the  mission  level.  A  discussion  of  system  design  and 
operational  test  and  evaluation  considerations  will  be  included.  Following  chapters 
will  focus  on  possible  models  for  linking  system  attributes  to  quantify  measures  of 
effectiveness  at  the  mission  level. 

B.  SYSTEM  EFFECTIVENESS  DEFINED 

Based  on  the  previous  chapters,  definitions  for  system  effectiveness  and 
measure  of  effectiveness  are  now  proposed: 

System  Effectiveness.  The  ability  of  a  system  to  accomplish  a  mission,  and 
achieve  a  favorable  battle  outcome,  in  an  optimum  manner. 

Measure  of  Effectiveness.  An  outcome-oriented  measure  of  a  system’s 
effectiveness. 

The  measurement  of  system  effectiveness  may  be  a  relative,  not  absolute, 
measure  of  a  system’s  utility  compared  to  other  systems.  However,  once  a  particular 
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system  concept  has  been  chosen  for  development,  system  effectiveness  must  be 
translated  into  absolute  threshold  values  that  provide  a  basis  upon  which  system 
attribute  specifications  can  be  developed.  Using  this  definition  of  system 
effectiveness,  this  chapter  will  provide  a  more  in-depth  discussion  of  how  system 
effectiveness  measurement  at  the  mission  level  relates  to  the  overall  system  design 
process. 

C.  THE  CHALLENGE  OF  SYSTEM  DESIGN 

1.  Relationship  Between  System  Attributes  and  System  Effectiveness 
Depending  on  the  system,  there  are  many  system  attributes  which  can 

potentially  contribute  to  system  effectiveness.  Keeping  in  mind  the  required 
relationship  between  measures  of  performance  (attributes)  and  measures  of 
effectiveness,  the  challenge  of  system  design  is  to  link  the  measures  of  the  many 
system  attributes  to  system  effectiveness.  This  is  accomplished  through  the  use  of 
tradeoff  analyses. 

2.  Tradeoff  Analyses 

The  current  official  guidance  concerning  tradeoff  analyses  (with  the  term 
"attribute"  inserted  as  applicable)  states: 

a.  Tradeoff  Analyses 

They  (tradeoff  analyses)  display  the  implications  of  "trading" 
one  set  of  controllable  variables  (such  as  system  attribute  measure¬ 
ments)  for  another  (such  as  cost)....  To  do  a  tradeoff  analysis,  one  must 
identify  areas  of  uncertainty,  conduct  sensitivity  analyses,  and  establish 
thresholds. 


(1)  Uncertainty.  Tradeoff  analyses  identify  areas  of 
uncertainty  and  estimate  their  extent.  The  implications  of  the 
uncertainties  are  examined  in  cost  models  and  effectiveness  models. 
This  serves  to  highlight  for  decision  makers  the  areas  in  which 
uncertainties  most  affect  the  analysis  and,  therefore,  its  results. 
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(2)  Sensitivity.  Sensitivity  analyses  show  explicitly 
how  military  utility  is  affected  by  changes  in  system  attributes.  They 
show  how  system  attributes  (size,  weight,  etc.)  drive  performance,  and 
how  performance  affects  military  utility  or  effectiveness.  Parameters 
should  be  varied  individually  where  it  is  reasonable  to  do  so.  The 
uncertainty  inherent  in  estimating  attributes  and  in  determining  their 
impact  should  be  portrayed  explicitly. 

(a)  As  a  result  of  this  step,  the  analysis  is  able 
to  show  "where  we  are  on  the  curve":  whether  the  desired  performance 
is  stretching  the  system  to  the  point  that  increases  in  performance  add 
little  of  benefit;  whether  the  results  are  sensitive  to  change. 

(b)  In  a  very  real  sense,  there  are  few  "hard, 
unchallengeable"  requirements  in  weapons  acquisition.  Certain 
attributes  and  levels  of  effectiveness  are  not  "essential,  regardless  of 
cost."  Sensitivity  analysis  illuminates  how  important  it  is  to 
incorporate  these  features  into  a  system. 

(3)  Thresholds.  An  important  step  in  developing  a 
cost  and  operational  effectiveness  analysis  is  to  determine  thresholds, 
the  maximum  cost  or  minimum  acceptable  performance  that  could  be 
tolerated  in  a  system.  In  order  to  approach  thresholds  and  acceptability 
thresholds  reasonably,  senior  decision  makers  and  users  must  be 
directly  involved  in  reviewing  the  combinations  of  cost  and 
performance  that  would  be  acceptable. 

(a)  Cost  thresholds  are  expressions  of  value. 
They  answer  questions  such  as:  How  valuable  is  a  given  capability  to 
the  Service?  How  much  would  a  Service  be  willing  to  give  up  in  order 
to  obtain  that  capability?  At  what  point  would  it  be  preferable  to  drop 
the  idea  in  favor  of  some  other  course  of  action? 

(b)  Performance  thresholds  may  be  more 
difficult  to  determine  but  are  at  least  as  important  as  cost  thresholds. 
They  show  at  what  point  degradations  in  performance  yield  outcomes 
that  no  longer  satisfy  the  mission  need.  Together,  cost  and  performance 
thresholds  help  in  determining  which  alternatives  are  worthwhile  and 
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what  combinations  or  intervals  of  performance  and  cost  are  acceptable. 

[Ref.  5:pp.  8-10  and  8-11] 

3.  Optimal  System  Design 

As  stated  earlier,  system  effectiveness  measurement  at  the  mission  level 
provides  the  most  direct  means  of  linking  system  attributes  to  system  effectiveness. 
Once  threshold  (minimum  acceptable)  values  for  system  effectiveness  at  the  mission 
level  have  been  identified,  system  engineers  are  faced  with  determining  the  optimal 
combination  of  system  attribute  specifications  (stated  as  thresholds  and  objectives) 
that  will  result  in  an  overall  effective,  and  affordable,  system.  Since  most  systems 
are  built  to  accomplish  multiple  missions,  and  most  missions  involve  numerous 
mission  tasks  (functional  objectives),  this  presents  a  dilemma  for  the  system  designer. 
Based  on  user  input  regarding  the  relative  importance  of  all  system  MOEs  to  overall 
system  effectiveness,  the  designer  determines  certain  system  attributes  which  become 
key  design  objectives  to  which  the  designer  optimizes  the  overall  system  design. 

D.  RELATING  MOES  TO  TEST  AND  EVALUATION  OF  MOPS 

1.  Operational  Test  and  Evaluation 

Once  the  engineers  have  developed  thresholds  and  objectives  for  system 
attributes,  the  system  is  built  to  the  desired  specifications.  Once  built,  the  system 
must  be  tested  and  evaluated  in  a  realistic  operational  setting  to  ensure  that  the  system 
attribute  specifications  are  achieved.  Also  at  this  stage,  the  attributes  should  be 
reevaluated  in  terms  of  their  impact  on  desired  system  effectiveness,  as  reflected  by 
the  COEA  MOEs. 

2.  The  1992  USD(A)  Memorandum 

The  importance  of  relating  measures  of  effectiveness  to  the  test  and  evaluation 
of  system  attributes  was  reemphasized  in  a  1992  Memorandum  from  the  Under 
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Secretary  of  Defense  for  Acquisition  [Ref.  13].  A  summary  of  the  guidance  (with  the 
term  "attribute"  inserted  where  applicable)  found  in  that  memorandum  is  included  in 
this  section. 

a.  The  Documents 

(1)  The  Cost  and  Operational  Effectiveness  Analysis 
(COEA).  The  DoD  component,  in  the  process  of  performing  a  Milestone  I  COEA, 
should  identify  the  MOEs  to  be  used  in  the  COEA  and  show  how  these  MOEs  are 
derived  from  the  Mission  Need  Statement  (MNS).  Each  COEA  should  include  MOEs 
reflecting  operational  utility  that  can  be  tested.  For  those  MOEs  that  cannot  be 
directly  tested,  the  COEA  should  show  how  changes  in  testable  attributes  or  MOPs 
can  be  related  to  changes  in  COEA  MOEs. 

(2)  The  Operational  Requirements  Document  (ORD).  The 
MOEs  and  related  MOPs  should  be  included  in  the  ORD. 

(3)  The  Acquisition  Program  Baseline  (APB).  The  key 
MOEs  and  MOPs  should  also  be  included  in  the  APB  subject  to  review  by  the 
Requirements  Validation  Authority  and  approval  by  the  Milestone  Decision 
Authority. 

(4)  Test  and  Evaluation  Master  Plan  (TEMP).  The  TEMP 
should  document  how  the  COEA  MOEs  and  related  MOPs  will  be  addressed  in  the 
test  and  evaluation  of  the  selected  alternative  system. 

b.  Linking  MOEs  to  Test  Results 

The  Milestone  Decision  Authority  and  the  Requirements  Validation 
Authority  should  be  able  to  review  the  COEA  using  test  results  to  reaffirm  the 
decision  that  the  selected  alternative  is  an  effective  approach  to  satisfying  an 
operational  requirement.  If  the  system  effectiveness  (MOE)  thresholds  stipulated  in 
the  APB  and  ORD  and  used  in  the  COEA  are  not  supported  by  test  results,  the  COEA 
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sensitivity  analyses  should  be  available  to  assist  in  determining  whether  the  system, 
as  tested,  still  offers  sufficient  military  benefit  to  be  worth  its  cost  and  whether  the 
system  can  be  confirmed  to  be  operational  effective  and  operationally  suitable. 

E.  SYSTEM  EFFECTIVENESS  MODELS 

The  necessity  for  measuring  system  effectiveness  in  terms  that  can  be  linked 
to  system  attributes  has  led  to  the  development  of  several  system  effectiveness 
models.  These  models  measure  system  effectiveness  in  terms  of  mission 
accomplishment  and  provide  frameworks  for  linking  system  attributes  to  system 
effectiveness.  In  the  next  chapter,  several  of  these  models  will  be  presented. 
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VI.  SEVERAL  SYSTEM  EFFECTIVENESS  MODELS 


A.  INTRODUCTION 

Ultimately,  the  developer  must  determine  the  individual  and  combined  effects 
of  all  system  attributes  when  choosing  between  alternative  systems  or  system  designs. 
In  Chapter  IV,  the  author  proposed  that  system  effectiveness  be  measured  at  both  the 
mission  and  battle  levels.  Of  these  two  levels,  the  mission  level  provides  the  most 
direct  means  for  relating  system  attributes  to  system  effectiveness.  This  argument  is 
supported  by  the  abundance  of  models  which  link  system  attributes  to  system 
effectiveness  at  the  mission  level,  and  the  lack  of  models  which  link  these  attributes 
to  system  effectiveness  at  the  battle  level.  This  chapter  will  describe  several  existing 
models  which  relate  mission-level  measures  of  effectiveness  to  combinations  of  key 
system  attributes. 

These  models  will  be  sub-divided  into  two  categories,  multiplicative  and 
additive.  The  multiplicative  models  are  the  Weapons  Systems  Effectiveness  Industry 
Advisory  Committee  (WSEIAC),  Habayeb,  Ball,  OPNAVINST  3000.12,  Marshall, 
and  Giordano  models.  Only  one  additive  model,  the  Georgia  Tech  Aerospace 
Systems  Design  Laboratory  (ASDL)  model,  will  be  described. 

B.  MULTIPLICATIVE  VERSUS  ADDITIVE  MODELS 

All  of  the  listed  models  develop  mission-specific  measures  of  effectiveness 
through  the  combination  of  a  few  key  system  attributes  into  mathematical  equations 
involving  attribute  measures.  There  are  two  basic  types  of  these  equations.  The  first, 
more  popular,  method  involves  the  multiplication  of  key  attribute  measures,  leading 
to  a  single  overall  measure  of  effectiveness.  The  second  method  involves  applying 
weighting  coefficients  to  various  key  attribute  measures  which  reflect  the  relative 
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importance  of  those  attributes,  and  then  adding  the  weighted  attribute  value,  leading 
to  a  single  overall  measure  of  effectiveness. 

C.  THE  MULTIPLICATIVE  MODELS 

The  multiplicative  models  which  follow  are  based  on  the  premise  that  a 
system's  effectiveness  in  accomplishing  a  particular  mission  is  a  product  of  a  few  key 
system  attributes.  These  attribute  measures  are  expressed  as  probabilities,  and  they 
all  must  be  present,  in  some  degree,  for  a  system  to  be  considered  effective.  The  most 
effective  system,  using  these  models,  is  the  one  with  the  highest  probability  of 
mission  accomplishment  over  time. 

1.  The  WSEIAC  Model 
a.  The  Equation 

In  the  final  Weapons  System  Effectiveness  Industry  Advisory 
Committee  (WSEIAC)  report  of  January  1965,  an  equation  identifying  three  key 
system  attributes,  Availability  (A),  Dependability  (D),  and  Capability  (C),  was 
presented  for  overall  system  effectiveness  (E)  evaluation: 

E  =  [A]  *  [D]  *  [C] 
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where  [E]  is  the  measure  of  the  extent  to  which  a  system  can  be  expected  to  achieve 
a  set  of  specific  mission  requirements  and  is  a  function  of  availability,  dependability, 
and  capability. 
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b.  Definitions  of  Key  Attributes 

(1)  Availability.  [A]  is  an  availability  row  matrix,  which 
presents  the  possible  states  of  system  condition  at  the  start  of  the  mission  and  is  a 
function  of  the  relationships  among  hardware,  personnel,  and  procedures. 

(2)  Dependability.  [D]  is  a  dependability  square  matrix, 
which  presents  the  probability  that  the  system  (1)  will  enter  and/or  occupy  any  one 
of  its  significant  states  during  the  specific  mission,  and  (2)  will  perform  the  functions 
associated  with  those  states. 

(3)  Capability.  [C]  is  a  capability  column  matrix,  which 
presents  the  ability  of  the  system  to  achieve  the  mission  objectives;  given  the  system 
condition(s)  during  the  mission,  and  specifically  accounts  for  the  performance 
spectrum  of  the  system. 

c.  Analysis 

This  model  relies  on  a  single-mission  scenario.  For  the  overall  mission 
envelope,  each  possible  system  state  is  described  and  a  matrix  developed  giving 
transition  probabilities  between  states.  The  product  of  the  three  matrices  is  equal  to 
overall  system  effectiveness.  Judgement  plays  an  integral  part  in  arriving  at 
capability  values  as  well  as  in  the  development  of  the  overall  operational  scenario  for 
which  the  dependability  matrix  is  set  up. 

2.  The  HABAYEB  Model 
a.  The  Equation 

Dr.  A.  R.  Habayeb,  the  technology  manager  for  weapons  command  and 
control  in  NAVAIR's  Weapons  Division  (AIR-912),  in  his  book  entitled  System 
Effectiveness  [Ref.  14],  proposes  an  equation  identifying  three  key  system  attributes, 
readiness  (sr),  reliability  (r),  and  design  adequacy  (da).  He  places  these  attributes  into 
an  analytical  structure  using  a  probabilistic  notion  of  system  effectiveness  (se), 
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Pse  =  Psr  *  Pr  *  Pda 


where  Pse  is  the  probability  that  the  system  is  effective. 

b.  Definitions  of  Key  Attributes 

(1)  Readiness  (Psr).  The  probability,  that,  at  any  point  in 
time,  the  total  system  is  ready  to  be  operated  on  demand  and  operates  satisfactorily 
when  used  under  specific  conditions. 

(2)  Reliability  (Pr).  The  probability  that  the  system  will 
operate  under  stated  conditions  without  malfunction  for  the  duration  of  the  mission, 
in  other  words,  the  mission  reliability  of  the  system. 

(3)  Design  Adequacy  (Pda).  The  probability  that  the  system 
will  successfully  accomplish  its  mission,  given  that  it  is  operated  within  design 
specification.  It  is  the  measure  for  performance  capability. 

c.  Hierarchy  of  A ttributes 

As  shown  in  Table  6. 1 ,  Habayeb  offers  a  hierarchy  in  which  lower-level 
attributes  are  listed  below  his  three  key  attributes: 


Readiness 

Design  Adequacy 

Transportability 

Reliability 

Survivability 

Reliability 

Durability 

Vulnerability 

Availability 

Quality 

Operational 

Suitability 

Supportability 

Interoperability 

Maintainability 

Compatibility 

Quality 

Durability 

Quality 

Table  6.1.  Habayeb’s  Hierarchy  of  System  Attributes 
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d.  Analysis 

Habayeb’s  equation  relies  on  a  single-mission  scenario.  His  three  key 
attributes  are  interdependent  operationally,  and  overall  system  effectiveness  requires 
they  all  occur.  The  attributes  are  usually  quantified  independently  of  each  other,  in 
steps  at  different  times.  Design  adequacy  (capability)  is  initially  built  into  the  system, 
and  is  fixed  until  changes  occur  in  the  environment,  e.g.,  new  application  or  threat. 
Therefore,  design  adequacy  is  initially  quantified  at  the  early  stages  of  system 
development,  determined  by  simulation  or  experimentally,  while  assuming  readiness 
and  reliability  events  occur  as  specified. 

Reliability  is  built  into  the  system  as  it  is  developed  and  is  quantified 
upon  development  of  a  system  block  diagram,  parts  counts  of  the  system,  and  the 
failure  rates  of  the  hardware.  Degradation  in  reliability  is  a  function  of  time  and  the 
environment  and  assumes  design  adequacy  and  readiness  are  as  specified. 

Lastly,  readiness  presents  procedural  (e.g.,  repair,  maintenance  and 
operational,  logistics),  and  training  events,  and  is  estimated  from  the  reliability 
assessment  and  the  mean  time  to  repair  (MTTR),  and  assumes  design  adequacy  is  as 
specified.  As  a  caveat,  Habayeb  states  that  some  aspects  of  reliability  (e.g., 
redundancy,  hardware  quality  and  testing)  and  readiness  (e.g.,  modular  design  and 
human  factors)  can  be  accounted  for  in  the  early  stages  of  design  adequacy.  System 
effectiveness  quantification  then  becomes  an  iterative  process,  where  operational 
feedback  data  is  used  to  continually  refine  the  quantification  of  the  three  key 
attributes. 

In  his  book,  Habayeb  has  developed  a  detailed  statistical  proof  of  his 
equation,  which  is  beyond  the  scope  of  this  thesis. 
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3.  The  BALL  Model 

a .  The  Equation 

Professor  Robert  E.  Ball,  a  faculty  member  of  the  Naval  Postgraduate 
School  Aeronautics  and  Astronautics  Department,  in  his  book,  The  Fundamentals  of 
Aircraft  Combat  Survivability  Analysis  and  Design  [Ref.  15],  proposes  a  measure  of 
combat  effectiveness  of  an  available  aircraft  in  a  particular  scenario  as  the  measure 
of  mission  success  (MOMS),  which  is  given  by 

MOMS  =  MAM  *  S 

b.  Definitions  of  Key  Attributes 

(1)  Mission  Attainment  Measure  (MAM).  The  mission 
attainment  measure  is  the  effectiveness  (expressed  as  a  range  between  0  and  1)  of 
the  offensive  capability  of  the  aircraft. 

(2)  Survivability  (S).  The  effectiveness  (also  expressed  as 
a  range  between  0  and  1 )  of  the  defense  capability  of  the  aircraft,  where 


Ps  =  1-Pk 

where  Ps  =  the  probability  of  survival,  and 
Pk  -  Ph*Pk/h 

Ph  =  the  probability  of  hit  (susceptibility)  and 

Pk/h  =  the  probability  of  kill  given  a  hit  (vulnerability) 

c.  Analysis 

The  MOMS  equation  was  solely  intended  to  provide  a  simple  measure 
of  effectiveness  for  system  operational  performance  in  a  combat  environment  and  the 
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impact  that  survivability  has  upon  mission  success.  Professor  Ball  uses  his  MOMS 
formula  to  illustrate  the  tradeoffs  between  MAM  and  S.  For  example,  in  many 
combat  situations  the  MAM  is  intentionally  reduced  in  order  to  increase  the  S.  In 
these  situations,  the  more  survivable  aircraft  can  possibly  have  a  smaller  MOMS  per 
aircraft.  This  translates  into  a  tradeoff  between  quality  (less  capability  per  aircraft) 
and  quantity  (more  survivable  aircraft)  to  achieve  the  same  overall  level  of  mission 
success. 

4.  The  OPNA VINST  3000. 12  Model 

a.  The  Equation 

The  OPNA  VINST  3000. 12,  Operational  Availability  of  Equipment  and 
Weapons  Systems  [Ref.  16],  provides  the  following  equation  which  provides  the  three 
key  system  attributes.  Operational  Capability,  Operational  Availability,  and 
Operational  Dependability,  as  key  to  determining  system  effectiveness  (SE): 

SE  =  Co  *  Ao  *  Do 

b.  Definitions  of  Key  Attributes 

(1)  Operational  Capability  (Co).  Refers  to  the  system's 
operating  characteristics  (range,  payload,  accuracy,  and  the  resultant  ability  to  counter 
the  threat,  in  terms  of  probability  of  kill,  exchange  ratios,  etc. 

(2)  Operational  Availability  (Ao).  Probability  that  a  system  is 
ready  to  perform  its  specified  function,  in  its  specific  operational  environment,  when 
called  upon  at  a  random  point  in  time. 

(3)  Operational  Dependability  (Do).  Probability  that  the 
system,  if  up  at  the  beginning  of  the  mission,  will  remain  up  throughout  the  mission. 
Also  referred  to  as  "mission  reliability". 
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c. 


Analysis 


The  OPNAVINST  3000.12  relies  on  a  single-mission  scenario  and  is 
intended  to  highlight  the  importance  of  operational  availability  as  a  critical  element 
of  system  effectiveness.  Other  than  offering  the  equation,  the  instruction  goes  no 
further  in  addressing  the  interrelationships  of  the  three  key  attributes  and  only  offers 
sub-elements  for  operational  availability.  Given  Ao  sub-elements  are  reliability, 
maintainability,  and  logistics  resources.  The  OPNAVINST  does  state  that  capability, 
availability,  and  dependability  must  be  defined  relative  to  the  specific  warfare 
environment  and  operating  scenario  envisioned  or  employed  for  a  given  system. 

5.  The  MARSHALL  Model 
a.  The  Equation 

John  Marshall,  retired  Marine  Corps  colonel,  and  systems  engineer  in 
the  Unmanned  Air  Vehicle  (UAV)  Project  Office,  Naval  Air  Warfare  Center  Aircraft 
Division  (NAWCAD)  in  his  ESP  Proposal  [Ref.  17],  has  proposed  an  equation  based 
on  those  of  Habayeb  and  Ball.  Marshall  combines  the  key  system  attributes, 
Operational  Availability  (Ao),  Mission  Reliability  (Rm),  Survivability  (S),  and 
Mission  Attainment  Measure  (MAM),  to  develop  a  single  measure  of  Effectiveness/ 
Suitability/  Performance  (ESP)  using  the  following  equation  for  system  effectiveness 
(SE): 


ESP  =  SE  =  Ao  *  Rm  *  S  *  MAM 

b.  Definitions  of  Key  Attributes 

(1)  Operational  Availability  (Ao).  (includes  logistics 
readiness)  The  probability  that  a  system  will  be  available  for  use  when  required. 
Given  by  equation:  [Total  up  time  /  (total  up  and  down  time)]  or  [#  of  ready  systems 
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/  total  #  systems].  Use  Inherent  Availability  (A,)  for  unfielded  systems.  -  readiness 
in  terms  of  inherent  design  characteristics.  Given  by  the  equation:  (MTBF)  / 
(MTBF+MTTR). 

(2)  Mission  Reliability  (Rm).  The  probability  that  a  system 
will  perform  mission  essential  functions  for  a  period  of  time  under  the  conditions 
stated  in  the  mission  profile.  Given  by  the  equation:  e(  T/MTBOMF) . 

(3)  Survivability  (S).  The  capability  of  a  system  to  avoid  or 
withstand  man-made  hostile  environments  without  suffering  an  abortive  impairment 
of  its  ability  to  accomplish  its  designated  mission.  Same  as  Ball’s  equation:  (1- 

Ph*Pk/h)- 

( 4)  Mission  Attainment  Measure  (MAM).  Mission  effective¬ 
ness.  Probability  of  performing  the  defined  mission,  given  Ao,  Rm,  and  S.  This 
probability  will  be  based  on  successive  trials  during  applicable  operational  scenarios. 
Given  by  the  equation  (1  -  error  rate). 

c.  Analysis 

The  ESP  model  relies  on  a  single-mission  scenario.  According  to 
Marshall,  the  model  provides  a  common  systems  evaluation  framework  for  DoD;  a 
quantifiable,  synergistic  means  to  define  and  evaluate  total  system  effectiveness.  The 
model  relies  on  a  cyclical  view  of  system  performance  and  the  key  attributes  that 
affect  performance.  The  ESP  model  does  not  obviate  the  need  for  detailed  analysis, 
but  rather,  it  incorporates  detailed  analyses  into  a  meaningful  top-level  equation  to  be 
used  by  decision  makers.  This  model  will  be  used  as  the  basis  for  developing  a 
system  attribute  hierarchy  in  the  next  chapter. 
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6. 


The  GIORDANO  Model 


a.  The  Basic  Equation 

Paul  Giordano,  Systems  Effectiveness  Branch  Head  of  the  U.S.  Naval 
Applied  Science  Laboratory,  in  his  1 966  report,  Predicting  Systems  Effectiveness 
[Ref.  18],  proposed  the  following  system  effectiveness  (E)  equation: 

E  =  Pc  *  Pt 

b.  Definitions  of  Key  Attributes 

(1)  Performance  Capability  (PJ  .  Giordano  does  not 
specifically  provide  a  definition.  However,  his  usage  of  the  term  corresponds  with 
the  description  of  operational  capability  given  in  the  OPNAVINST  model.  Thus, 
performance  capability  refers  to  the  system's  operating  characteristics  (range,  payload, 
accuracy)  and  the  resultant  ability  to  counter  the  threat,  in  terms  of  probability  of  kill, 
exchange  ratios,  etc. 

(2)  Time  Dependency  of  Performance  (P).  Resource  penalty 
parameters.  Adds  a  time  dimension  to  performance  and  allows  all  the  "ilities"  to  be 
treated  as  modifiers  of  optimal  performance. 

c.  Analysis  of  Basic  Equation 

Giordano’s  basic  equation  is  designed  for  a  single-mission  scenario. 
According  to  Giordano,  the  program  manager  will  build  a  system  effectiveness  model 
based  on  specific  attribute  requirements,  based  on  factors  related  to  mission 
accomplishment.  The  sequence  of  events  is  given  below. 

First,  the  mission  is  reduced  to  tasks,  which,  if  accomplished,  will  give 
the  program  manager  confidence  of  mission  success.  The  task  descriptions  form  the 
basis  for  Effectiveness  Analysis.  Next,  attribute  requirements  are  established  that 
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allow  successful  task  accomplishment.  The  attribute  values  are  varied  according  to 
task  requirements.  Acceptable  levels  of  attribute  values  then  become  measures  for 
detailed  analytical  modeling  to  assess  sensitivities  to  resources,  failures,  logistics, 
degradation,  and  time  demands.  The  interplay  of  simulation  and  analytical  modeling 
is  used  to  set  up  and  evaluate  systems  and  specific  designs  within  systems.  Giordano 
sees  his  equation  as  a  means  of  treating  all  chosen  attributes  in  a  unified  fashion  and 
treating  resource  penalty-related  attributes  as  an  integral  part  of  overall  system 
performance. 

Giordano's  envisions  that  the  program  manager  should  treat  resource 
penalty  attributes  in  three  distinct  ways.  First,  absolute  limits,  or  "thresholds"  are 
placed  on  some  penalties,  and  treated  as  "go,  no  go"  requirements.  Second,  the 
penalties  are  treated  as  design  parameters  and  their  impact  is  measured  where  possible 
as  a  change  in  the  equation.  Finally,  the  penalties  are  treated  as  quantitative  attributes 
of  a  system  in  their  own  right  and  specific  values  are  estimated. 

This  leads  Giordano  into  a  discussion  of  combining  attributes  into  an 
overall  equation.  If  one  can  put  a  relative  value  on  the  attributes,  they  can  be 
combined  into  a  figure  of  merit  or  measure  of  effectiveness.  However,  Giordano 
warns  against  combining  data  beyond  the  point  where  it  causes  loss  of  information 
at  the  expense  of  simplicity.  This  is  due  to  the  variability  built  into  the  analysis. 

d.  The  Multimission  Equation 

Of  all  the  multiplicative  models,  only  Giordano  discusses  system 
effectiveness  for  multimission  scenarios.  Giordano  expands  his  basic  equation  to 
portray  a  sample  representation  of  system  effectiveness  for  a  large  system  in  a 
multimission  scenario  as  follows: 

E  =  W(Pc*Pt)/LT,  W(Pc*Pt)/CA,  W(Pc*Pt)/MP,  \V(Pc*Pt)/CO 
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where  LT  is  Lead  Time,  CA  is  Acquisition  Cost,  MP  is  Manpower,  CO  is 
Ownership  Cost,  and  W  is  a  measure  of  mission  worth. 

e.  Analysis  of  Multimission  Equation 

The  factors  LT,  CA,  MP,  and  CO  are  examples  of  potential  modifiers 
of  the  basic  equation.  The  factor  W  is  used  to  allow  additional  variation  where  multi¬ 
missions  are  treated  independently  or  combined.  W  allows  iterative  analysis  per 
mission. 

Note  that  one  could  translate  Giordano’s  multi-mission  equation  into 
an  additive  equation,  where  the  factor  W  can  be  used  to  separately  weight  single¬ 
mission  measures,  and  then  combine  them  into  an  overall  measure  of  multimission 
system  effectiveness. 

D.  THE  ADDITIVE  MODEL  -  ASDL 
1.  The  Equation 

Dr.  D.  N.  Mavris  and  Mr.  D.  DeLaurentis  from  the  Aerospace  Systems  Design 
Labaratory  at  the  School  of  Aerospace  Engineering,  Georgia  Institute  of  Technology, 
in  their  paper  entitled  "An  Integrated  Approach  to  Military  Aircraft  Selection  and 
Concept  Evaluation"  [Ref.  19],  presented  an  equation  identifying  five  key  attributes, 
Affordability,  Survivability,  Readiness,  Capability,  and  Safety,  which  combine  into 
a  single,  overall  measure  of  an  aircraft  system  effectiveness,  or  overall  evaluation 
criterion  (OEC): 

OEC  =  a(/LCC/LCCBL)  +  b(MCI/MCIBL)  +  c(EAI/EAIBL)  +  d(Psurv/PsurvBL)  + 
e(Aj/AiBL) 
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where  a  through  e  are  importance  coefficients  which  sum  to  1,  LCC  is  Life  Cycle 
Cost,  MCI  is  Mission  Capability  Index,  EAI  is  Engine  Related  Attrition  Index,  Psurv 
is  a  measure  of  Survivability,  and  A;  is  Inherent  Availability. 

2.  Definitions  of  Key  Attributes 

a.  Life  Cycle  Cost  (LCC).  A  overall  measurement  of 
Affordability,  or  the  total  cost  to  the  government  for  acquisition,  ownership, 
operation,  and  disposal  of  the  system  over  its  life  cycle. 

b.  Mission  Capability  Index  (MCI).  A  overall  measurement  of 
Capability,  or  the  aircraft’s  ability  to  complete  its  mission  (satisfy  or  exceed  all 
mission  requirements). 

c.  Engine-Caused  Attrition  Index  (EAI).  A  overall  measurement 
of  Safety,  or  an  estimation  of  the  effect  on  peacetime  attrition  of  engine  induced 
Class-A  failures  based  on  total  number  of  aircraft  operated. 

d.  Probability  of  Survival  (Psurv).  An  overall  measurement  of 
Survivability,  or  an  aircraft’s  ability  to  evade  detection  and  avoid  damage  which 
would  result  in  loss  of  vehicle. 

e.  Inherent  Availability  (Aft  An  overall  measurement  of 
Readiness,  or  the  degree  to  which  an  item  is  in  operable  and  committal  state  at  the 
start  of  the  mission  when  the  mission  is  called  for,  at  an  unknown  time. 

3.  Analysis 

The  ASDL  equation  relies  on  a  single-mission  scenario  and  is  used  as  follows. 
In  comparing  the  effectiveness  of  competing  systems,  one  picks  a  system,  assigns  a 
value  of  "1"  to  all  the  factors  of  that  system,  and  then  describes  competing  systems 
relative  to  the  base  system.  For  example,  if  the  baseline  system  A  has  a  .90 
survivability,  then  system  B  with  a  .45  survivability  would  be  assigned  a  MOE  for 
that  factor  of  .5.  Subsequently,  a  weighting  coefficient  is  applied  to  survivability  to 
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show  how  important  it  is  to  the  customer.  This  is  repeated  for  each  attribute. 
Summation  of  the  five  weighted  attributes  into  a  top-level  measure,  as  is  shown  in  the 
ASDL  equation,  provides  a  comparison  of  an  alternative  system's  overall  effective¬ 
ness  to  the  baseline  system  and  is  helpful  in  choosing  between  alternative  systems. 

One  problem  with  the  ASDL  equation  is  the  subjectivity  associated  with 
assigning  weighting  coefficients.  As  the  coefficients  are  basically  measures  of 
human  preferences,  the  resultant  OEC  becomes  a  measure  of  preference,  rather  than 
effectiveness.  Depending  on  one’s  preferences,  choices  between  systems  may 
change.  However,  the  system  itself,  and  its  ability  to  accomplish  the  mission,  may 
remain  the  same.  In  fact,  if  one  has  a  predisposed  preference  for  one  system  over  all 
the  others,  one  may  be  able  to  manipulate  the  coefficients  until  the  equation  shows  the 
preferred  system  to  be  the  most  effective.  Thus,  the  objectivity  of  this  model  may  be 
quite  low. 

E.  SINGLE  VERSUS  MULTIPLE  MISSIONS 

For  the  most  part,  the  above  equations  only  provide  a  measurement  of  system 
effectiveness  in  accomplishing  a  specific  mission  in  a  given  operational  scenario. 
Furthermore,  in  order  to  use  these  equations,  one  must  define  mission 
accomplishment  using  a  single  number.  The  basic  operational  capability  (or  design 
adequacy,  MAM,  MCI)  is  then  either  degraded  by  other  parameters,  in  the 
multiplicative  equations,  or  weighted  against  other  parameters,  in  the  additive 
equation.  Most  systems  today  are  built  to  accomplish  multiple  missions  in  multiple 
operational  scenarios.  Thus,  using  these  equations,  different  numbers  would  have  to 
be  derived  for  each  new  mission  and  scenario,  resulting  in  multiple  measures  of 
effectiveness.  Giordano  addresses  the  multimission  scenario  by  suggesting  a  possible 
weighting  of  missions  which  would  produce  an  overall  system  effectiveness  number. 
The  current  DoDI  5000.2  warns  against  such  weighting  schemes: 
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Never  use  schemes  in  which  several  measures  of  effectiveness  are 
weighted  and  combined  into  an  overall  score.  Weighting  schemes  are 
sometimes  helpful,  but  they  must  be  clearly  explained  in  the  analysis 
so  that  their  results  can  be  interpreted  correctly.  [Ref.  6:p.  8-12] 

Thus,  these  equations  may  only  be  useful  in  providing  a  single-number  of  system 
effectiveness  for  a  specific  mission,  and  measuring  overall  system  effectiveness 
would  still  depend  on  the  subjective  combination  of  several  measures  of  effectiveness 
for  multiple  missions. 

F.  COMPARISON  OF  MODELS 
1.  The  Multiplicative  Models 

These  models,  with  the  exception  of  the  aircraft-specific  Ball  model,  are  all 
purported  to  provide  a  standard  means  of  measuring  system  effectiveness  for  any 
system.  The  Giordano  model  addresses  system  effectiveness  in  the  simplest  terms, 
combining  performance  capability  (Pc)  with  only  one  other  attribute  (jP),  which 
signifies  the  numerous  factors  which  can  degrade  performance  over  time.  The 
Marshall  model,  on  the  other  hand,  provides  the  most  complex  approach,  adding  the 
Mission  Attainment  Measure  (MAM)  and  Survivability  (S)  attributes  from  Ball’s 
model  to  the  availability  (Ao)  and  reliability  (Rm)  attributes  of  the  WSEIAC, 
Habayeb,  and  OPNAVINST  models.  Only  the  Giordano  model  includes  a  discussion 
of  measuring  system  effectiveness  beyond  a  mission-specific  scenario. 

All  of  the  multiplicative  models  present  system  effectiveness  as  some  form  of 
operational  capability  which  is  degraded  over  mission  time.  The  multiplicative 
models  use  a  cycle  of  mission  accomplishment,  which  over  time  provides  a  measure 
of  a  system's  effectiveness  in  accomplishing  a  particular  mission.  Each  system 
attribute  is  probabilistic  and  through  direct  multiplication,  combines  equally  with 
other  attributes  resulting  in  a  measure  of  effectiveness  between  0  and  1.  One 
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difficulty  in  using  these  models  is  the  measurement  of  a  system's  operational 
capability  as  a  number  between  0  and  1 .  Furthermore,  the  relevance  of  multiplying 
the  key  system  attribute  measurements  into  a  single-number  measure  of  effectiveness 
is  yet  unproven. 

2.  The  Additive  Models 

The  ASDL  model  presents  system  effectiveness  for  an  aircraft  as  the  result  of 
a  variety  of  weighted  attributes.  Like  some  of  the  multiplicative  models,  this  model 
includes  availability,  survivability,  and  a  mission  capability  index  (operational 
capability).  Unlike  the  multiplicative  models,  however,  the  additive  model  also 
includes  cost  and  safety  as  attributes  which  lead  to  system  effectiveness.  This  model 
relies  on  assigning  importance  coefficients  to  each  attribute,  which  then  combine  into 
an  overall  measure  of  effectiveness.  The  additive  approach  will  not  be  discussed 
further  in  this  thesis. 

3.  A  Modified  Approach 

In  the  next  chapter,  the  author  will  provide  a  more  detailed  discussion  of  the 
cycle  of  mission  accomplishment  and  develop  a  system  attribute  hierarchy  which  may 
be  useful  in  relating  the  numerous  system  attributes  to  key  attributes  and  hence  to 
system  effectiveness. 
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VII.  A  SYSTEM  ATTRIBUTE  HIERARCHY 


A.  INTRODUCTION 

The  current  official  guidance  does  not  provide  a  hierarchy  or  tree  for  relating 
system  measures  of  performance  (attribute  measures)  to  overall  system  effectiveness. 
Thus,  there  is  no  standard  process  by  which  the  numerous  attributes  which  make  up 
a  system  are  cohesively  linked  to  system  effectiveness.  In  the  previous  chapter,  the 
author  presented  several  models  which  link  key  system  attributes  to  system 
effectiveness,  resulting  in  a  single-number  measure  of  system  effectiveness  in 
accomplishing  a  particular  mission.  Although  the  relevance  of  combining  the 
numerical  values  of  system  attributes  into  a  single-number  measure  of  system 
effectiveness  is  yet  unproven,  an  understanding  of  the  relationship  between  system 
attributes  and  system  effectiveness  is  essential.  Accordingly,  the  author  will  now 
present  the  beginning  of  a  hierarchy  or  tree  which  relates  many  system  attributes  to 
four  key  attributes.  Measures  for  each  of  the  attributes  are  suggested.  This  new 
hierarchy  will  be  closely  related  to  the  Marshall  Model  (see  Chapter  VI),  where 
system  effectiveness  is  defined  as  the  result  of  a  cycle  of  mission  accomplishment 
(attributes).  Any  measure  of  system  effectiveness  in  accomplishing  a  particular 
mission  will  most  likely  use  these  four  key  attributes.  The  hierarchy  is  proposed  as 
a  tool  which  might  assist  system  engineers,  analysts,  and  test  and  evaluation  personnel 
in  relating  tradeoffs  between  the  numerous  system  attributes  to  mission-level 
measures  of  effectiveness. 

B.  THE  CYCLE  OF  MISSION  ACCOMPLISHMENT 

1.  The  Key  System  Attributes 

Describing  the  combined  effect  of  all  attributes  on  overall  system  effectiveness 
(mission  accomplishment)  over  the  life  of  a  system  is  quite  cumbersome  if  stated  in 
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terms  of  the  separate  effect  of  each  attribute.  A  more  useful  hierarchy  of  attributes 
can  be  developed  through  an  understanding  of  the  cycle  of  mission  accomplishment. 
Shown  in  Figure  7.1,  this  cycle  identifies  four  overall  key  system  attributes  which 
describe  overall  system  performance  over  time.  For  some  systems,  the  cycle  of 
mission  accomplishment  may  occur  only  once.  However,  for  most  systems,  the  cycle 
may  be  repeated  many  times.  Although  the  order  of  attribute  occurrence  may  vary, 
the  cycle  usually  begins  with  a  system  being  available  for  use  (Availability)  on  a 
given  mission,  continues  with  the  system  not  breaking  down  during  the  mission  due 
to  malfunction  (Reliability)  or  being  killed  or  critically  damaged  by  a  threat 
(Survivability),  and  ends  with  some  form  of  mission  accomplishment  (Capability). 


Figure  7.1.  Cycle  of  Mission  Accomplishment 
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2.  The  Key  System  Attributes  Defined 

Use  of  this  model  necessitates  the  following  definitions,  and  measurements, 
for  the  given  key  system  attributes: 

a.  Availability 

The  availability  of  a  fielded  system,  called  Operational  Availability, 
defined  as  the  probability  that  a  system  will  be  available  for  use  when  required. 
Measurements  can  be  either  [Total  up  time  /  (total  up  and  down  time)]  or  [#  of  ready 
systems  /  total  #  systems].  For  an  unfielded  system,  called  Inherent  Availability, 
defined  as  readiness  in  terms  of  inherent  design  characteristics.  Measurement  is 
(Mean  Time  Between  Failure)  /  (Mean  Time  Between  Failure  +Mean  Time  To 
Repair). 

b.  Reliability 

The  probability  that  a  system  will  perform  mission  essential  functions 
for  a  period  of  time  under  the  conditions  stated  in  the  mission  profile.  Measurement 

js  g(-T/  MTBOMF) 

c.  Survivability 

The  capability  of  a  system  to  avoid  or  withstand  hostile  environments 
without  suffering  an  abortive  impairment  of  its  ability  to  accomplish  the  designated 
mission.  Measurement  is  Ps. 

d.  Capability 

A  measure  of  mission  accomplishment  (i.e.,  mission  completion  time, 
#  of  messages  sent,  #  of  enemy  tanks  killed)  in  a  particular  operational  scenario. 

3.  The  Importance  of  Time 

The  above  model  provides  an  overall  description  of  key  system  attributes  in 
terms  of  the  cyclical  employment  of  a  system  in  its  intended  operational  environment 
over  time.  The  element  of  time  is  important,  as  most  military  missions  are  part  of 
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larger  operational  scenarios  which  involve  repeated  system  employment,  at  a  planned 
utilization  rate,  over  a  specified  time  period.  The  cycle  must  be  assessed  over  time, 
as  a  proper  measure  of  overall  system  effectiveness  must  consider  the  time-related 
degradation  in  a  system’s  ability  to  accomplish  a  mission.  This  holds  true  even  for 
fire-and-forget  systems  such  as  missiles,  where  repeated  firings  are  usually  necessary 
in  the  overall  accomplishment  of  a  mission. 

C.  A  PROPOSED  SYSTEM  ATTRIBUTE  HIERARCHY 

A  system  attribute  hierarchy  or  tree  can  now  be  developed  by  expanding 
beyond  the  four  key  attributes  of  the  cycle  of  mission  accomplishment  to  include 
consideration  of  lower-level  attributes.  This  hierarchy,  shown  in  Figure  7.2,  provides 
a  method  of  relating  tradeoffs  between  the  numerous  system  attributes  to  overall 
system  effectiveness. 

D.  CONCLUSION 

Using  the  proposed  cycle  of  mission  accomplishment,  a  relationship  between 
a  system’s  numerous  attributes  and  system  effectiveness  at  the  mission  level  is 
developed.  This  relationship  is  shown  in  the  proposed  system  attribute  hierarchy. 
The  tree  consists  of  four  kev  attributes,  Availability,  Reliability,  Survivability,  and 
Capability,  which  result  in  system  effectiveness  in  accomplishing  a  particular  mission. 
Although  the  relevance  of  combining  the  four  key  system  attribute  values  into  an 
single-number  system  effectiveness  measurement  is  yet  unproven,  the  hierarchy  itself 
provides  a  useful  tool  for  understanding  and  conducting  tradeoffs  between  various 
system  attributes  in  arriving  at  an  optimal  level  of  system  effectiveness. 
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Figure  7.2.  System  Attribute  Hierarchy 
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VIII.  SUMMARY  AND  CONCLUSIONS 


In  the  system  acquisition  process,  the  effectiveness  and  the  cost  of  a  system  are 
the  two  key  ingredients  in  determining  system  value  to  the  military  services.  The 
effectiveness  of  a  system  is  directly  related  to  the  characteristics  or  attributes  of  that 
system  (e.g.,  speed,  reliability,  survivability).  Success  in  system  acquisition  relies  on 
(1)  the  early  identification  and  successful  incorporation  of  those  system  attributes  that 
are  critical  to  system  effectiveness,  and  (2)  the  specification  of  numerical  values  for 
the  system  attributes  (the  system  requirements)  that  maximizes  system  effectiveness 
at  an  acceptable  cost. 

In  this  thesis,  the  relationship  between  the  attributes,  or  characteristics,  of  a 
military  weapon  system  and  system  effectiveness  of  that  system  was  thoroughly 
examined.  The  current  guidance  mandates  that  system  effectiveness  be  measured  in 
terms  of  engagement  or  battle  outcomes  (at  the  battle  level).  To  provide  a  basis  for 
linking  system  attributes  to  system  effectiveness,  the  author  provided  a  wide  spectrum 
of  system  acquisition-related  documentation  which  supports  an  additional  viewpoint 
to  the  current  DoDI  5000.2  -  that  system  effectiveness  should  also  be  measured  in 
terms  of  mission  outcomes  (at  the  mission  level).  Furthermore,  in  the  system 
development  and  test  process,  the  mission  level  provides  the  best  means  for  linking 
system  attribute  measurements  to  system  effectiveness  measurement.  Accordingly, 
the  author  proposed  new  definitions  for  system  effectiveness  and  measures  of 
effectiveness  which  acknowledge  that  system  effectiveness  should  be  measured  at 
both  the  mission  and  battle  levels. 

The  current  DoDI  5000.2  guidance  also  does  not  provide  a  standard  model  for 
linking  system  attributes  to  system  effectiveness.  Thus,  the  author  described  several 
existing  multiplicative  and  additive  models  which  combine  a  few  key  system  attribute 
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measurements  into  single-number  measures  of  system  effectiveness  in  accomplishing 
a  particular  mission.  Finally,  using  a  similar  basis  to  the  multiplicative  models,  the 
cycle  of  mission  accomplishment,  the  author  proposed  a  hierarchy  or  tree  which 
relates  many  system  attributes  to  the  four  key  attributes,  Availability,  Reliability, 
Survivability,  and  Capability,  and  hence  to  system  effectiveness  in  accomplishing  a 
specified  mission.  Although  the  author  acknowledged  the  need  for  a  model  relating 
system  attributes  to  system  effectiveness,  the  relevance  of  developing  single-number 
measures  of  system  effectiveness  was  left  unproven.  Ultimately,  the  goal  of  this 
thesis  was  to  provide  the  reader  with  a  better  understanding  of  the  relationship 
between  system  attributes  and  system  effectiveness. 
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